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Foreword 


Big data is transforming the traditional ways of handling data to make sense of the 
world from which it is collected. Statisticians, for instance, are used to developing 
methods for analysis of data collected for a specific purpose in a planned way. 
Sample surveys and design of experiments are typical examples. 

Big data, in contrast, refers to massive amounts of very high dimensional and 
even unstructured data which are continuously produced and stored with much 
cheaper cost than they are used to be. High dimensionality combined with large 
sample size creates unprecedented issues such as heavy computational cost and 
algorithmic instability. 

The massive samples in big data are typically aggregated from multiple sources 
at different time points using different technologies. This can create issues of 
heterogeneity, experimental variations, and statistical biases, and would therefore 
require the researchers and practitioners to develop more adaptive and robust 
procedures. 

Toward this, I am extremely happy to see in this title not just a compilation of 
chapters written by international experts who work in diverse disciplines involving 
Big Data, but also a rare combination, within a single volume, of cutting-edge work 
in methodology, applications, architectures, benchmarks, and data standards. 

I am certain that the title, edited by three distinguished experts in their fields, will 
inform and engage the mind of the reader while exploring an exciting new territory 
in science and technology. 


Calyampudi Radhakrishna Rao 

C.R. Rao Advanced Institute of Mathematics, 
Statistics and Computer Science, 

Hyderabad, India 


www.allitebooks.com 


Preface 


The emergence of the field of Big Data Analytics has prompted the practitioners 
and leaders in academia, industry, and governments across the world to address and 
decide on different issues in an increasingly data-driven manner. Yet, often Big 
Data could be too complex to be handled by traditional analytical frameworks. The 
varied collection of themes covered in this title introduces the reader to the richness 
of the emerging field of Big Data Analytics in terms of both technical methods as 
well as useful applications. 

The idea of this title originated when we were organizing the “Statistics 2013, 
International Conference on Socio-Economic Challenges and Sustainable Solutions 
(STAT2013)” at the C.R. Rao Advanced Institute of Mathematics, Statistics and 
Computer Science (AIMSCS) in Hyderabad to mark the “International Year of 
Statistics” in December 2013. As the convener, Prof. Saumyadipta Pyne organized 
a special session dedicated to lectures by several international experts working on 
large data problems, which ended with a panel discussion on the research chal- 
lenges and directions in this area. Statisticians, computer scientists, and data ana- 
lysts from academia, industry and government administration participated in a 
lively exchange. 

Following the success of that event, we felt the need to bring together a col- 
lection of chapters written by Big Data experts in the form of a title that can 
combine new algorithmic methods, Big Data benchmarks, and various relevant 
applications from this rapidly emerging area of interdisciplinary scientific pursuit. 
The present title combines some of the key technical aspects with case studies and 
domain applications, which makes the materials more accessible to the readers. In 
fact, when Prof. Pyne taught his materials in a Master’s course on “Big and 
High-dimensional Data Analytics” at the University of Hyderabad in 2013 and 
2014, it was well-received. 
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We thank all the authors of the chapters for their valuable contributions to this 
title. Also, We sincerely thank all the reviewers for their valuable time and detailed 
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Big Data Analytics: Views from Statistical 
and Computational Perspectives 


Saumyadipta Pyne, B.L.S. Prakasa Rao and S.B. Rao 


Abstract Without any doubt, the most discussed current trend in computer science 
and statistics is BIG DATA. Different people think of different things when they hear 
about big data. For the statistician, the issues are how to get usable information out of 
datasets that are too huge and complex for many of the traditional or classical meth- 
ods to handle. For the computer scientist, big data poses problems of data storage 
and management, communication, and computation. For the citizen, big data brings 
up questions of privacy and confidentiality. This introductory chapter touches some 
key aspects of big data and its analysis. Far from being an exhaustive overview of 
this fast emerging field, this is a discussion on statistical and computational views 
that the authors owe to many researchers, organizations, and online sources. 


1 Some Unique Characteristics of Big Data 


Big data exhibits a range of characteristics that appears to be unusual when com- 
pared to traditional datasets. Traditionally, datasets were generated upon conscious 
and careful planning. Field experts or laboratory experimenters typically spend con- 
siderable time, energy, and resources to produce data through planned surveys or 
designed experiments. However, the world of big data is often nourished by dynamic 
sources such as intense networks of customers, clients, and companies, and thus there 
is an automatic flow of data that is always available for analysis. This almost volun- 
tary generation of data can bring to the fore not only such obvious issues as data 
volume, velocity, and variety but also data veracity, individual privacy, and indeed, 
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ethics. If data points appear without anticipation or rigor of experimental design, 
then their incorporation in tasks like fitting a suitable statistical model or making a 
prediction with a required level of confidence, which may depend on certain assump- 
tions about the data, can be challenging. On the other hand, the spontaneous nature 
of such real-time pro-active data generation can help us to capture complex, dynamic 
phenomena and enable data-driven decision-making provided we harness that ability 
in a cautious and robust manner. For instance, popular Google search queries could 
be used to predict the time of onset of a flu outbreak days earlier than what is pos- 
sible by analysis of clinical reports; yet an accurate estimation of the severity of the 
outbreak may not be as straightforward [1]. A big data-generating mechanism may 
provide the desired statistical power, but the same may also be the source of some 
its limitations. 

Another curious aspect of big data is its potential of being used in unintended 
manner in analytics. Often big data (e.g., phone records) could be used for the type 
of analysis (say, urban planning) that is quite unrelated to the original purpose of its 
generation, especially if the purpose is integration or triangulation of diverse types 
of data, including auxiliary data that may be publicly available. If a direct survey of 
a society’s state of well-being is not possible, then big data approaches can still pro- 
vide indirect but valuable insights into the society’s socio-economic indicators, say, 
via people’s cell phone usage data, or their social networking patterns, or satellite 
images of the region’s energy consumption or the resulting environmental pollution, 
and so on. Not only can such unintended usage of data lead to genuine concerns 
about individual privacy and data confidentiality, but it also raises questions regard- 
ing enforcement of ethics on the practice of data analytics. 

Yet another unusual aspect that sometimes makes big data what it is is the ratio- 
nale that if the generation costs are low, then one might as well generate data on as 
many samples and as many variables as possible. Indeed, less deliberation and lack of 
parsimonious design can mark such “glut” of data generation. The relevance of many 
of the numerous variables included in many big datasets seems debatable, especially 
since the outcome of interest, which can be used to determine the relevance of a 
given predictor variable, may not always be known during data collection. The actual 
explanatory relevance of many measured variables to the eventual response may be 
limited (so-called “variable sparsity”), thereby adding a layer of complexity to the 
task of analytics beyond more common issues such as data quality, missing data, and 
spurious correlations among the variables. 

This brings us to the issues of high variety and high dimensionality of big data. 
Indeed, going beyond structured data, which are “structured” in terms of variables, 
samples, blocks, etc., and appear neatly recorded in spreadsheets and tables result- 
ing from traditional data collection procedures, increasingly, a number of sources 
of unstructured data are becoming popular—text, maps, images, audio, video, news 
feeds, click streams, signals, and so on. While the extraction of the essential fea- 
tures can impose certain structure on it, unstructured data nonetheless raises con- 
cerns regarding adoption of generally acceptable data standards, reliable annotation 
of metadata, and finally, robust data modeling. Notably, there exists an array of pow- 
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erful tools that is used for extraction of features from unstructured data, which allows 
combined modeling of structured and unstructured data. 

Let us assume a generic dataset to be a n X p matrix. While we often refer to big 
data with respect to the number of data points or samples therein (denoted above 
by n), its high data volume could also be due to the large number of variables 
(denoted by p) that are measured for each sample in a dataset. A high-dimensional 
or “big p” dataset (say, in the field of genomics) can contain measurements of tens 
of thousands of variables (e.g., genes or genomic loci) for each sample. Increasingly, 
large values of both p and n are presenting practical challenges to statisticians and 
computer scientists alike. High dimensionality, 1.e., big p relative to low sample size 
or small n, of a given dataset can lead to violation of key assumptions that must be 
satisfied for certain common tests of hypotheses to be applicable on such data. In 
fact, some domains of big data such as finance or health do even produce infinite 
dimensional functional data, which are observed not as points but functions, such as 
growth curves, online auction bidding trends, etc. 

Perhaps the most intractable characteristic of big data is its potentially relentless 
generation. Owing to automation of many scientific and industrial processes, it is 
increasingly feasible, sometimes with little or no cost, to continuously generate dif- 
ferent types of data at high velocity, e.g., streams of measurements from astronom- 
ical observations, round-the-clock media, medical sensors, environmental monitor- 
ing, and many “big science” projects. Naturally, if streamed out, data can rapidly gain 
high volume as well as need high storage and computing capacity. Data in motion can 
neither be archived in bounded storage nor held beyond a small, fixed period of time. 
Further, it is difficult to analyze such data to arbitrary precision by standard iterative 
algorithms used for optimal modeling or prediction. Other sources of intractability 
include large graph data that can store, as network edges, static or dynamic informa- 
tion on an enormous number of relationships that may exist among individual nodes 
such as interconnected devices (the Internet of Things), users (social networks), com- 
ponents (complex systems), autonomous agents (contact networks), etc. To address 
such variety of issues, many new methods, applications, and standards are currently 
being developed in the area of big data analytics at a rapid pace. Some of these have 
been covered in the chapters of the present title. 


2 Computational versus Statistical Complexity 


Interestingly, the computer scientists and the statisticians—the two communities of 
researchers that are perhaps most directly affected by the phenomenon of big data— 
have, for cultural reasons, adopted distinct initial stances in response to it. The pri- 
mary concern of the computer scientist—who must design efficient data and file 
structures to store massive datasets and implement algorithms on them—stems from 
computational complexity. It concerns the required number of computing steps to 
solve a given problem whose complexity is defined in terms of the length of input 
data as represented by a reasonable encoding mechanism (say, a N bit binary string). 
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Therefore, as data volume increases, any method that requires significantly more 
than O(N log(N)) steps (1.e., exceeding the order of time that a single pass over the 
full data would require) could be impractical. While some of the important prob- 
lems in practice with O(N log(N)) solutions are just about scalable (e.g., Fast Fourier 
transform), those of higher complexity, certainly including the NP-Complete class 
of problems, would require help from algorithmic strategies like approximation, ran- 
domization, sampling, etc. Thus, while classical complexity theory may consider 
polynomial time solutions as the hallmark of computational tractability, the world 
of big data is indeed even more demanding. 

Big data are being collected in a great variety of ways, types, shapes, and sizes. 
The data dimensionality p and the number of data points or sample size n are usually 
the main components in characterization of data volume. Interestingly, big p small 
n datasets may require a somewhat different set of analytical tools as compared to 
big n big p data. Indeed, there may not be a single method that performs well on all 
types of big data. Five aspects of the data matrix are important [2]: 


(1) the dimension p representing the number of explanatory variables measured; 
(11) the sample size n representing the number of observations at which the variables 
are measured or collected; 

(11i) the relationship between n and p measured by their ratio; 

(iv) the type of variables measured (categorical, interval, count, ordinal, real-valued, 
vector-valued, function-valued) and the indication of scales or units of measure- 
ment; and 

(v) the relationship among the columns of the data matrix to check multicollinearity 
in the explanatory variables. 


To characterize big data analytics as different from (or extension of) usual data 
analysis, one could suggest various criteria, especially if the existing analytical 
strategies are not adequate for the solving the problem in hand due to certain proper- 
ties of data. Such properties could go beyond sheer data volume. High data velocity 
can present unprecedented challenges to a statistician who may not be used to the 
idea of forgoing (rather than retaining) data points, as they stream out, in order to 
satisfy computational constraints such as single pass (time constraint) and bounded 
storage (space constraint). High data variety may require multidisciplinary insights 
to enable one to make sensible inference based on integration of seemingly unrelated 
datasets. On one hand, such issues could be viewed merely as cultural gaps, while 
on the other, they can motivate the development of the necessary formalisms that 
can bridge those gaps. Thereby, a better understanding of the pros and cons of dif- 
ferent algorithmic choices can help an analyst decide about the most suitable of the 
possible solution(s) objectively. For instance, given a p variable dataset, a time-data 
complexity class can be defined in terms of n(p), r(p) and t(p) to compare the perfor- 
mance tradeoffs among the different choices of algorithms to solve a particular big 
data problem within a certain number of samples n(p), a certain level of error or risk 
r(p) and a certain amount of time f(p) [3]. 

While a computer scientist may view data as physical entity (say a string hav- 
ing physical properties like length), a statistician is used to viewing data points 
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as instances of an underlying random process for data generation, typically mod- 
eled using suitable probability distributions. Therefore, by assuming such underlying 
structure, one could view the growing number of data points as a potential source 
of simplification of that structural complexity. Thus, bigger n can lead to, in a clas- 
sical statistical framework, favorable conditions under which inference based on the 
assumed model can be more accurate, and model asymptotics can possibly hold [3]. 
Similarly, big p may not always be viewed unfavorably by the statistician, say, if the 
model-fitting task can take advantage of data properties such as variable sparsity 
whereby the coefficients—say, of a linear model—corresponding to many variables, 
except for perhaps a few important predictors, may be shrunk towards zero [4]. In 
particular, it is the big p and small n scenario that can challenge key assumptions 
made in certain statistical tests of hypotheses. However, while data analytics shifts 
from a static hypothesis-driven approach to a more exploratory or dynamic large 
data-driven one, the computational concerns, such as how to decide each step of an 
analytical pipeline, of both the computer scientist and the statistician have gradually 
begun to converge. 

Let us suppose that we are dealing with a multiple linear regression problem with 
p explanatory variables under Gaussian error. For a model space search for variable 
selection, we have to find the best subset from among 2? — 1 sub-models. If p = 20, 
then 2? — 1 is about a million; but if p = 40, then the same increases to about a 
trillion! Hence, any problem with more than p = 50 variables is potentially a big data 
problem. With respect to n, on the other hand, say, for linear regression methods, it 
takes O(n?) number of operations to invert an n x n matrix. Thus, we might say that 
a dataset is big n if n > 1000. Interestingly, for a big dataset, the ratio n/p could 
be even more important than the values of n and p taken separately. According to a 
recent categorization [2], it is information-abundant if n/p > 10, information-scarce 
if 1 < n/p < 10, and information-poor if n/p < 1. 

Theoretical results, e.g., [5], show that the data dimensionality p is not consid- 
ered as “big” relative to n unless p dominates s/n asymptotically. If p >> n, then there 
exists a multiplicity of solutions for an optimization problem involving model-fitting, 
which makes it ill-posed. Regularization methods such as the Lasso (cf. Tibshirani 
[6]) are used to find a feasible optimal solution, such that the regularization term 
offers a tradeoff between the error of the fit model and its complexity. This brings us 
to the non-trivial issues of model tuning and evaluation when it comes to big data. A 
‘“model-complexity ladder” might be useful to provide the analyst with insights into 
a range of possible models to choose from, often driven by computational considera- 
tions [7]. For instance, for high-dimensional data classification, the modeling strate- 
gies could range from, say, naïve Bayes and logistic regression and moving up to, 
possibly, hierarchical nonparametric Bayesian approaches. Ideally, the decision to 
select a complex model for a big dataset should be a careful one that is justified by 
the signal-to-noise ratio of the dataset under consideration [7]. 

A more complex model may overfit the training data, and thus predict poorly for 
test data. While there has been extensive research on how the choice of a model with 
unnecessarily high complexity could be penalized, such tradeoffs are not quite well 
understood for different types of big data, say, involving streams with nonstationary 
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characteristics. If the underlying data generation process changes, then the data com- 
plexity can change dynamically. New classes can emerge or disappear from data (also 
known as “concept drift”) even while the model-fitting is in progress. In a scenario 
where the data complexity can change, one might opt for a suitable nonparametric 
model whose complexity and number of parameters could also grow as more data 
points become available [7]. For validating a selected model, cross-validation is still 
very useful for high-dimensional data. For big data, however, a single selected model 
does not typically lead to optimal prediction. If there is multicollinearity among the 
variables, which is possible when p is big, the estimators will be unstable and have 
large variance. Bootstrap aggregation (or bagging), based on many resamplings of 
size n, can reduce the variance of the estimators by aggregation of bootstrapped ver- 
sions of the base estimators. For big n, the “bag of small bootstraps” approach can 
achieve similar effects by using smaller subsamples of the data. It is through such 
useful adaptations of known methods for “small” data that a toolkit based on fun- 
damental algorithmic strategies has now evolved and is being commonly applied to 
big data analytics, and we mention some of these below. 


3 Techniques to Cope with Big Data 


Sampling, the general process of selecting a subset of data points from a given input, 
is among the most established and classical techniques in statistics, and proving to 
be extremely useful in making big data tractable for analytics. Random sampling 
strategies are commonly used in their simple, stratified, and numerous other variants 
for their effective handling of big data. For instance, the classical Fisher—Yates shuf- 
fling is used for reservoir sampling in online algorithms to ensure that for a given 
“reservoir” sample of k points drawn from a data stream of big but unknown size 
n, the probability of any new point being included in the sample remains fixed at 
k/n, irrespective of the value of the new data. Alternatively, there are case-based 
or event-based sampling approaches for detecting special cases or events of interest 
in big data. Priority sampling is used for different applications of stream data. The 
very fast decision tree (VFDT) algorithm allows big data classification based on a 
tree model that is built like CART but uses subsampled data points to make its deci- 
sions at each node of the tree. A probability bound (e.g., the Hoeffding inequality) 
ensures that had the tree been built instead using the full dataset, it would not differ 
by much from the model that is based on sampled data [8]. That is, the sequence of 
decisions (or “splits’) taken by both trees would be similar on a given dataset with 
probabilistic performance guarantees. Given the fact that big data (say, the records 
on the customers of a particular brand) are not necessarily generated by random 
sampling of the population, one must be careful about possible selection bias in the 
identification of various classes that are present in the population. 

Massive amounts of data are accumulating in social networks such as Google, 
Facebook, Twitter, LinkedIn, etc. With the emergence of big graph data from social 
networks, astrophysics, biological networks (e.g., protein interactome, brain connec- 
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tome), complex graphical models, etc., new methods are being developed for sam- 
pling large graphs to estimate a given network’s parameters, as well as the node-, 
edge-, or subgraph-statistics of interest. For example, snowball sampling is a com- 
mon method that starts with an initial sample of seed nodes, and in each step i, it 
includes in the sample all nodes that have edges with the nodes in the sample at step 
i — | but were not yet present in the sample. Network sampling also includes degree- 
based or PageRank-based methods and different types of random walks. Finally, the 
Statistics are aggregated from the sampled subnetworks. For dynamic data streams, 
the task of statistical summarization is even more challenging as the learnt mod- 
els need to be continuously updated. One approach is to use a “sketch”, which is 
not a sample of the data but rather its synopsis captured in a space-efficient rep- 
resentation, obtained usually via hashing, to allow rapid computation of statistics 
therein such that a probability bound may ensure that a high error of approximation 
is unlikely. For instance, a sublinear count-min sketch could be used to determine 
the most frequent items in a data stream [9]. Histograms and wavelets are also used 
for statistically summarizing data in motion [8]. 

The most popular approach for summarizing large datasets is, of course, cluster- 
ing. Clustering is the general process of grouping similar data points in an unsu- 
pervised manner such that an overall aggregate of distances between pairs of points 
within each cluster is minimized while that across different clusters is maximized. 
Thus, the cluster-representatives, (say, the k means from the classical k-means clus- 
tering solution), along with other cluster statistics, can offer a simpler and cleaner 
view of the structure of the dataset containing a much larger number of points 
(n > k) and including noise. For big n, however, various strategies are being used 
to improve upon the classical clustering approaches. Limitations, such as the need 
for iterative computations involving a prohibitively large O(n?) pairwise-distance 
matrix, or indeed the need to have the full dataset available beforehand for conducting 
such computations, are overcome by many of these strategies. For instance, a two- 
step online—offline approach (cf. Aggarwal [10]) first lets an online step to rapidly 
assign stream data points to the closest of the k’ (« n) “microclusters.” Stored in 
efficient data structures, the microcluster statistics can be updated in real time as 
soon as the data points arrive, after which those points are not retained. In a slower 
offline step that is conducted less frequently, the retained k’ microclusters’ statistics 
are then aggregated to yield the latest result on the k(< k’) actual clusters in data. 
Clustering algorithms can also use sampling (e.g., CURE [11]), parallel computing 
(e.g., PKMeans [12] using MapReduce) and other strategies as required to handle 
big data. 

While subsampling and clustering are approaches to deal with the big n prob- 
lem of big data, dimensionality reduction techniques are used to mitigate the chal- 
lenges of big p. Dimensionality reduction is one of the classical concerns that can 
be traced back to the work of Karl Pearson, who, in 1901, introduced principal com- 
ponent analysis (PCA) that uses a small number of principal components to explain 
much of the variation in a high-dimensional dataset. PCA, which is a lossy linear 
model of dimensionality reduction, and other more involved projection models, typ- 
ically using matrix decomposition for feature selection, have long been the main- 
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stays of high-dimensional data analysis. Notably, even such established methods can 
face computational challenges from big data. For instance, O(n?) time-complexity 
of matrix inversion, or implementing PCA, for a large dataset could be prohibitive 
in spite of being polynomial time—1.e. so-called “tractable’’—solutions. 

Linear and nonlinear multidimensional scaling (MDS) techniques—working with 
the matrix of O(n’)  pairwise-distances between all data points in 
a high-dimensional space to produce a low-dimensional dataset that preserves the 
neighborhood structure—also face a similar computational challenge from big ndata. 
New spectral MDS techniques improve upon the efficiency of aggregating a global 
neighborhood structure by focusing on the more interesting local neighborhoods 
only, e.g., [13]. Another locality-preserving approach involves random projections, 
and is based on the Johnson—Lindenstrauss lemma [14], which ensures that data 
points of sufficiently high dimensionality can be “embedded” into a suitable low- 
dimensional space such that the original relationships between the points are approx- 
imately preserved. In fact, it has been observed that random projections may make 
the distribution of points more Gaussian-like, which can aid clustering of the pro- 
jected points by fitting a finite mixture of Gaussians [15]. Given the randomized 
nature of such embedding, multiple projections of an input dataset may be clustered 
separately in this approach, followed by an ensemble method to combine and pro- 
duce the final output. 

The term “curse of dimensionality” (COD), originally introduced by R.E. Bell- 
man in 1957, is now understood from multiple perspectives. From a geometric per- 
spective, as p increases, the exponential increase in the volume of a p-dimensional 
neighborhood of an arbitrary data point can make it increasingly sparse. This, in turn, 
can make it difficult to detect local patterns in high-dimensional data. For instance, 
a nearest-neighbor query may lose its significance unless it happens to be limited to 
a tightly clustered set of points. Moreover, as p increases, a “deterioration of expres- 
siveness” of the L, norms, especially beyond L, and L,, has been observed [16]. A 
related challenge due to COD is how a data model can distinguish the few relevant 
predictor variables from the many that are not, 1.e., under the condition of dimension 
sparsity. If all variables are not equally important, then using a weighted norm that 
assigns more weight to the more important predictors may mitigate the sparsity issue 
in high-dimensional data and thus, in fact, make COD less relevant [4]. 

In practice, the most trusted workhorses of big data analytics have been parallel 
and distributed computing. They have served as the driving forces for the design of 
most big data algorithms, softwares and systems architectures that are in use today. 
On the systems side, there is a variety of popular platforms including clusters, clouds, 
multicores, and increasingly, graphics processing units (GPUs). Parallel and distrib- 
uted databases, NoSQL databases for non-relational data such as graphs and docu- 
ments, data stream management systems, etc., are also being used in various applica- 
tions. BDAS, the Berkeley Data Analytics Stack, is a popular open source software 
stack that integrates software components built by the Berkeley AMP Lab (and third 
parties) to handle big data [17]. Currently, at the base of the stack, it starts with 
resource virtualization by Apache Mesos and Hadoop Yarn, and uses storage sys- 
tems such as Hadoop Distributed File System (HDFS), Auxilio (formerly Tachyon) 
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and Succinct upon which the Spark Core processing engine provides access and 
interfaces to tasks like data cleaning, stream processing, machine learning, graph 
computation, etc., for running different applications, e.g., cancer genomic analysis, 
at the top of the stack. 

Some experts anticipate a gradual convergence of architectures that are designed 
for big data and high-performance computing. Important applications such as large 
simulations in population dynamics or computational epidemiology could be built 
on top of these designs, e.g., [18]. On the data side, issues of quality control, stan- 
dardization along with provenance and metadata annotation are being addressed. On 
the computing side, various new benchmarks are being designed and applied. On 
the algorithmic side, interesting machine learning paradigms such as deep learning 
and advances in reinforcement learning are gaining prominence [19]. Fields such 
as computational learning theory and differential privacy will also benefit big data 
with their statistical foundations. On the applied statistical side, analysts working 
with big data have responded to the need of overcoming computational bottlenecks, 
including the demands on accuracy and time. For instance, to manage space and 
achieve speedup when modeling large datasets, a “chunking and averaging” strategy 
has been developed for parallel computation of fairly general statistical estimators 
[4]. By partitioning a large dataset consisting of n 1.1.d. samples (into r chunks each 
of manageable size |n/r|), and computing the estimator for each individual chunk 
of data in a parallel process, it can be shown that the average of these chunk-specific 
estimators has comparable statistical accuracy as the estimate on the full dataset [4]. 
Indeed, superlinear speedup was observed in such parallel estimation, which, as n 
grows larger, should benefit further from asymptotic properties. 


4 Conclusion 


In the future, it is not difficult to see that perhaps under pressure from 
the myriad challenges of big data, both the communities—of computer scientists and 
statistics—may come to share mutual appreciation of the risks, benefits and tradeoffs 
faced by each, perhaps to form a new species of data scientists who will be better 
equipped with dual forms of expertise. Such a prospect raises our hopes to address 
some of the “giant” challenges that were identified by the National Research Council 
of the National Academies in the United States in its 2013 report titled ‘Frontiers in 
Massive Data Analysis’. These are (1) basic statistics, (2) generalized N-body prob- 
lem, (3) graph-theoretic computations, (4) linear algebraic computations, (5) opti- 
mization, (6) integration, and (7) alignment problems. (The reader is encouraged to 
read this insightful report [7] for further details.) The above list, along with the other 
present and future challenges that may not be included, will continue to serve as a 
reminder that a long and exciting journey lies ahead for the researchers and practi- 
tioners in this emerging field. 
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Abstract In this study, we provide an overview of the state-of-the-art technologies 
in programming, computing, and storage of the massive data analytics landscape. 
We shed light on different types of analytics that can be performed on massive data. 
For that, we first provide a detailed taxonomy on different analytic types along with 
examples of each type. Next, we highlight technology trends of massive data ana- 
lytics that are available for corporations, government agencies, and researchers. In 
addition, we enumerate several instances of opportunities that exist for turning mas- 
sive data into knowledge. We describe and position two distinct case studies of mas- 
sive data analytics that are being investigated in our research group: recommendation 
systems in e-commerce applications; and link discovery to predict unknown associ- 
ation of medical concepts. Finally, we discuss the lessons we have learnt and open 
challenges faced by researchers and businesses in the field of massive data analytics. 
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1 Introduction 


1.1 Motivation 


Growth of Internet usage in the last decade has been at an unprecedented rate from 16 
million, which is about 0.4 % of total population in 1995, to more than 3 billion users, 
which is about half of the world’s population in mid-2014. This revolutionized the 
way people communicate and share their information. According to [46], just during 
2013, 4.4 zettabytes (4.4 x 2”? bytes) of information have created and replicated, and 
it estimated to grow up to 44 zettabytes by 2020. Below, we explain few sources from 
such massive data generation. 

Facebook! has an average of 1.39 billion monthly active users exchanging billions 
of messages and postings every day [16]. There is also a huge surge in multimedia 
content like photos and videos. For example, in popular photo sharing social net- 
work Instagram,” on average, 70 million photos uploaded and shared every day [27]. 
According to other statistics published by Google on its video streaming service, 
YouTube,” has approximately 300 h of video uploaded every minute and billions of 
views generated every day [62]. 

Along with Individuals, organizations are also generating a huge amount of data, 
mainly due to increased use of networked sensors in various sectors of organizations. 
For example, by simply replacing traditional bar code systems with radio frequency 
identification (RFID) systems organizations have generated 100 to 1000 times more 
data [57]. 

Organization’s interest on customer behavior is another driver for producing mas- 
sive data. For instance, Wal-Mart* handles more than a million customer transactions 
each hour and maintains a database that holds more than 2.5 petabytes of data [57]. 
Many businesses are creating a 360° view of a customer by combining transaction 
data with social networks and other sources. 

Data explosion is not limited to individuals or organizations. With the increase of 
scientific equipment sensitivity and advancements in technology, the scientific and 
research, community is also generating a massive amount of data. Australian Square 
Kilometer Array Pathfinder radio telescope [8] has 36 antennas streams approxi- 
mately 250 GB of data per second per antenna that collectively produces nine ter- 
abytes of data per second. In another example, particle accelerator, particle detector, 
and simulations at Large Hadron Collider (LHC) at CERN [55] generate approxi- 
mately 15 petabytes of data per year. 


"https://facebook.com. 
*https://instagram.com. 
>http://www.youtube.com. 
*http://www.walmart.com. 
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1.2 Big Data Overview 


The rapid explosion of data is usually referred as “Big Data”, which is a trending 
topic in both industry and academia. Big data (aka Massive Data) is defined as, data 
that cannot be handled or analyzed by conventional processing and storage tools. 
Big data is also characterized by features,known as 5V’s. These features are: volume, 
variety, velocity, variability, and veracity [7, 21]. 

Traditionally, most of the available data is structured data and stored in conven- 
tional databases and data warehouses for supporting all kinds of data analytics. With 
the Big data, data is no longer necessarily structured. Instead, it contains a variety of 
data sources, including structured, semi-structured, and unstructured data [7]. It is 
estimated that 85 % of total organizational data are unstructured data [57] and almost 
all the data generated by individuals (e.g., emails, messages, blogs, and multime- 
dia) are unstructured data too. Traditional relational databases are no longer a viable 
option to store text, video, audio, images, and other forms of unstructured data. This 
creates a need for special types of NoSQL databases and advanced analytic methods. 

Velocity of data is described as problem of handling and processing data at the 
speeds at which they are generated to extract a meaningful value. Online retailers 
store every attribute (e.g., clicks, page visits, duration of visits to a page) of their 
customers’ visits to their online websites. There is a need to analyze customers’ visits 
within a reasonable timespan (e.g., real time) to recommend similar items and related 
items with respect to the item a customer is looking at. This helps companies to 
attract new customers and keep an edge over their competitors. Some organizations 
analyze data as a stream in order to reduce data storage. For instance, LHC at CERN 
[55] analyzes data before storing to meet the storage requirements. Smart phones are 
equipped with modern location detection sensors that enable us to understand the 
customer behavior while, at the same time, creating the need for real-time analysis 
to deliver location-based suggestions. 

Data variability is the variation in data flow with time of day, season, events, etc. 
For example, retailers sell significantly more in November and December compared 
to rest of year. According to [1], traffic to retail websites surges during this period. 
The challenge, in this scenario, is to provide resources to handle sudden increases in 
users’ demands. Traditionally, organizations were building in-house infrastructure to 
support their peak-estimated demand periods. However, it turns out to be costly, as 
the resources will remain idle during the rest of the time. However, the emergence of 
advanced distributed computing platforms, known as ‘the cloud,’ can be leveraged 
to enable on-demand resource provisioning through third party companies. Cloud 
provides efficient computational, storage, and other services to organizations and 
relieves them from the burden of over-provisioning resources [49]. 

Big data provides advantage in decision-making and analytics. However, among 
all data generated in 2013 only 22 % of data are tagged, or somehow characterized 
as useful data for analysis, and only 5% of data are considered valuable or “Tar- 
get Rich” data. The quality of collected data, to extract a value from, is referred as 
veracity. The ultimate goal of an organization in processing and analyzing data is 


14 M.K. Pusala et al. 


to obtain hidden information in data. Higher quality data increases the likelihood 
of effective decision-making and analytics. A McKinsey study found that retailers 
using full potential from Big data could increase the operating margin up to 60 % 
[38]. To reach this goal, the quality of collected data needs to be improved. 


1.3 Big Data Adoption 


Organizations have already started tapping into the potential of Big data. Conven- 
tional data analytics are based on structured data, such as the transactional data, that 
are collected in a data warehouse. Advanced massive data analysis helps to com- 
bine traditional data with data from different sources for decision-making. Big data 
provides opportunities for analyzing customer behavior patterns based on customer 
actions inside (e.g., organization website) and outside (e.g., social networks). 

In a manufacturing industry, data from sensors that monitor machines’ operation 
are analyzed to predict failures of parts and replace them in advance to avoid sig- 
nificant down time [25]. Large financial institutions are using Big data analytics to 
identify anomaly in purchases and stop frauds or scams [3]. 

In spite of the wide range of emerging applications for Big data, organizations are 
still facing challenges to adopt Big data analytics. A report from AIIM [9], identified 
three top challenges in the adoption of Big data, which are lack of skilled workers, 
difficulty to combine structured and unstructured data, and security and privacy con- 
cerns. There is a sharp rise in the number of organizations showing interest to invest 
in Big data related projects. According to [18], in 2014, 47 % of organizations are 
reportedly investing in Big data products, as compared to 38 % in 2013. IDC pre- 
dicted that the Big data service market has reached 11 billion dollars in 2009 [59] 
and it could grow up to 32.4 billion dollars by end of 2017 [43]. Venture capital fund- 
ing for Big data projects also increased from 155 million dollars in 2009 to more than 
893 million dollars in 2013 [59]. 


1.4 The Chapter Structure 


From the late 1990s, when Big data phenomenon was first identified, until today, 
there has been many improvements in computational capabilities, storage devices 
have become more inexpensive, thus, the adoption of data-centric analytics has 
increased. In this study, we provide an overview of Big data analytic types, offer 
insight into Big data technologies available, and identify open challenges. 

The rest of this paper is organized as following. In Sect. 2, we explain different 
categories of Big data analytics, along with application scenarios. Section 3 of the 
chapter describes Big data computing platforms available today. In Sect. 4, we pro- 
vide some insight into the storage of huge volume and variety data. In that section, we 
also discuss some commercially available cloud-based storage services. In Sect. 5, 
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we present two real-world Big data analytic projects. Section 6 discusses open chal- 
lenges in Big data analytics. Finally, we summarize and conclude the main contri- 
butions of the chapter in Sect. 7. 


2 Big Data Analytics 


Big data analytics is the process of exploring Big data, to extract hidden and valu- 
able information and patterns [48]. Big data analytics helps organizations in more 
informed decision-making. Big data analytics applications can be broadly classi- 
fied as descriptive, predictive, and prescriptive. Figure 1 illustrates the data analytic 
classes, techniques, and example applications. In the rest of this section, with refer- 
ence to Fig. 1, we elaborate on these Big data analytic types. 


2.1 Descriptive Analytics 


Descriptive analytics mines massive data repositories to extract potential patterns 
existing in the data. Descriptive analytics drills down into historical data to detect 
patterns like variations in operating costs, sales of different products, customer buy- 
ing preferences, etc. 

Typically it is the first step of analytics in decision-making, answering the ques- 
tion of “what has happened? ”. It summarizes raw data into a human understandable 
format. Most of the statistical analysis used in day-to-day Business Intelligence (BI) 
regarding a company’s production, financial operations, sales, inventory, and cus- 
tomers come under descriptive analytics [61]. Analytics involve simple techniques, 
such as regression to find correlation among various variables and drawing charts, 


Reg ression Based on last year’s sales, how much profit this year? 
Descriptive- 


Visualization /nteractive dashboards in BI 


Forecasting Forecasting companies performance 
Big data analytics Predictive rin Predicting failure of machines 


Scoring Financial credit score for companies or individuals 


Optimization Optimize energy consumption and raw material in manufacturing 
Prescriptiv Numeric Modeling Control oil drilling model using internal and external variables 
Simulation Simulation scenarios in disaster management 
Fig. 1 Types of Big data analytics: The second level in the hierarchy is the categorization of ana- 


lytics. The third level, explains the typical techniques, and provides example in the corresponding 
analytic category 
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to identify trends in the data, and visualize data in a meaningful and understandable 
way, respectively. 

For example, Dow Chemicals used descriptive analytics to identify under-utilized 
space in its offices and labs. As a result, they were able to increase space utilization 
by 20 % and save approximately $4 million annually [14]. 


2.2 Predictive Analytics 


With descriptive analytics, organizations can understand what happened in the past. 
However, at a higher level of decision-making is to address the question of “what 
could happen?”. Predictive analytics helps to combine massive data from different 
sources with the goal of predicting future trends or events. Predictive analytics eval- 
uates the future, by forecasting trends, by generating prediction models, and by scor- 
ing. 

For example, industries use predictive analytics to predict machine failures using 
streaming sensor data [25]. Organizations are able to forecast their sales trends or 
overall performance [35]. Financial institutions devote a lot of resources to predict 
credit risk scores for companies or individuals. Eventhough predictive analytics can- 
not predict with 100% certainty, but it helps the companies in estimating future 
trends for more informed decision-making. 

Southwest airlines has partnered with National Aeronautics and Space Adminis- 
tration (NASA) to work on a Big data-mining project [42]. They apply text-based 
analysis on data from sensors in their planes in order to find patterns that indicate 
potential malfunction or safety issues. 

Purdue University uses Big data analytics to predict academic and behavioral 
issues [45]. For each student, the system predicts and generates a risk profile indi- 
cating how far a student succeeds in a course and labels the risk levels as green (high 
probability of success), yellow (potential problems), and red (risk of failure) by using 
data from various sources, such as student information and course management sys- 
tems for this analytics. 

E-commerce applications apply predictive analytics on customer purchase his- 
tory, customer behavior online, like page views, clicks, and time spend on pages, 
and from other sources [10, 58]. Retail organizations are able to predict customer 
behavior to target appropriate promotions and recommendations [31]. They use pre- 
dictive analysis to determine the demand of inventory and maintain the supply chain 
accordingly. Predictive analysis also helps to change price dynamically to attract 
consumers and maximize profits [2]. 
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2.3 Prescriptive Analytics 


Descriptive and predictive analytics helps to understand the past and predict the 
future. The next stage in decision-making is “how can we make it happen?” —the 
answer is prescriptive analytics. The goal of prescriptive analytics is to assist profes- 
sionals in assessing the impact of different possible decisions. It is a relatively new 
analytic method. According to Gartner [19], only 3 % of companies use prescriptive 
analytics in their decision-making. Prescriptive analytics involves techniques such 
as optimization, numerical modeling, and simulation. 

Oil and Gas exploration industries use prescriptive analytics to optimize the 
exploration process. Explorers are using massive datasets from different sources in 
the exploration process and use prescriptive analytics to optimize drilling location 
[56]. They use earth’s sedimentation characteristics, temperature, pressure, soil type, 
depth, chemical composition, molecular structures, seismic activity, machine data, 
and others to determine the best possible location to drill [15, 17]. This helps to 
optimize selection of drilling location, and avoid the cost and effort of unsuccessful 
drills. 

Health care is one of the sectors benefiting from applying Big data prescriptive 
analytics. Prescriptive analytics can recommend diagnoses and treatments to a doc- 
tor by analyzing patient’s medical history, similar conditioned patient’s history, aller- 
gies, medicines, environmental conditions, stage of cure, etc. According to [54], the 
Aurora Health Care Center saves six million USD annually by using Big data ana- 
lytics and recommending best possible treatment to doctors. 


3 Big Data Analytics Platforms 


There are several Big data analytics platforms available. In this section, we present 
advances within the Big data analytics platforms. 


3.1 MapReduce 


MapReduce framework represents a pioneering schema for performing Big data ana- 
lytics. It has been developed for a dedicated platform (such as a cluster). MapReduce 
framework has been implemented in three different ways. The first implementation 
was achieved by Google [13] under a proprietary license. The other two implemen- 
tations are: Hadoop [33] and Spark [66],which are available as open source. There 
are other platforms that, in fact, stem from these basic platforms. 

The core idea of MapReduce is based on developing two input functions namely, 
Map and Reduce. Programmers need to implement these functions. Each of these 
functions utilizes the available resources to process Big data in parallel. The MapRe- 
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duce works closely with a distributed storage system to carry out operations such as 
storing input, intermediate, and output data. Distributed file systems, such as Hadoop 
Distributed File System (HDFS) [52] and Google File System (GFS), have been 
developed to the MapReduce framework [20]. 

Every MapReduce workflow typically contains three steps (phases) namely, Map- 
ping step, Shuffling step, and Reduce step. In the Map step, user (programmer) imple- 
ments the functionality required in the Map function. The defined Map function will 
be executed against the input dataset across the available computational resources. 
The original (1.e., input) data are partitioned and placed in a distributed file system 
(DFS). Then, each Map task processes a partition of data from the DFS and gen- 
erates intermediate data that are stored locally on the worker machines where the 
processing was taking place. 

Distributing the intermediate data on the available computational resources is 
required to enable parallel Reduce. This step is known as Shuffling. The distribution 
of the intermediate data is performed in an all-to-all fashion that generally creates a 
communication bottleneck. Once the distribution of intermediate data is performed, 
the Reduce function is executed to produce the output, which is the final result of the 
MapReduce processing. Commonly, developers create a chain of MapReduce jobs 
(also referred to as a multistage MapReduce job), such as the Yahoo! WebMap [5]. 
In this case, the output of one MapReduce job is consumed as the intermediate data 
for the next MapReduce job in the chain. 


3.2 Apache Hadoop 


Hadoop [33] framework was developed as an open source product by Yahoo! and 
widely adopted for Big data analytics by the academic and industrial communities. 
The main design advantage of Hadoop 1s its fault-tolerance. In fact, Hadoop has been 
designed with the assumption of failure as a common issue in distributed systems. 
Therefore, it is robust against failures commonly occur during different phases of 
execution. 

Hadoop Distributed File System (HDFS) and MapReduce are two main building 
blocks of Hadoop. The former is the storage core of Hadoop (see Sect. 4.1 for details). 
The latter, MapReduce engine, is above the file system and takes care of executing 
the application by moving binaries to the machines that have the related data. 

For the sake of fault-tolerance, HDFS replicates data blocks in different racks; 
thus, in case of failure in one rack, the whole process would not fail. A Hadoop cluster 
includes one master node and one or more worker nodes. The master node includes 
four components namely, JobTracker, TaskTracker, NameNode, and DataNode. The 
worker node just includes DataNode and TaskTracker. The JobTracker receives user 
applications and allocates them to available TaskTracker nodes, while considering 
data locality. JobTracker assures about the health of TaskTrackers based on regu- 
lar heartbeats it receives from them. Although Hadoop is robust against failures in 
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a distributed system, its performance is not the best amongst other available tools 
because of frequent disk accesses [51]. 


3.3 Spark 


Spark is a more recent framework developed at UC Berkeley [66]. It is being used 
for research and production applications. Spark offers a general-purpose program- 
ming interface in the Scala programming language for interactive, in-memory data 
analytics of large datasets on a cluster. 

Spark provides three data abstractions for programming clusters namely, resilient 
distributed datasets (RDDs), broadcast variables, and accumulators. RDD 1s a read- 
only collection of objects partitioned across a set of machines. It can reconstruct lost 
partitions or recover in the event of a node failure. RDD uses a restricted shared 
memory to achieve fault-tolerance. Broadcast variables and accumulators are two 
restricted types of shared variables. Broadcast variable is a shared object wrapped 
around a read-only value, which ensures it is only copied to each worker once. Accu- 
mulators are shared variables with an add operation. Only workers can perform an 
operation on an accumulator and only users’ driver programs can read from it. Even- 
though, these abstractions are simple and limited, they can be used to develop several 
cluster-based applications. 

Spark uses master/slave architecture. It has one master instance, which runs a 
user-defined driver program. At run-time, the driver program launches multiple 
workers in the cluster, which read data from the shared filesystem (e.g., Hadoop 
Distributed File System). Workers create RDDs and write partitions on RAM as 
defined by the driver program. Spark supports RDD transformations (e.g., map, fil- 
ter) and actions (e.g., count, reduce). Transformations generate new datasets and 
actions return a value, from the existing dataset. 

Spark has proved to be 20X faster than Hadoop for iterative applications, was 
shown to speed up a real-world data analytics report by 40X, and has been used 
interactively to scan a | TB dataset with 57 s latency [65]. 


3.4 High Performance Computing Cluster 


LexisNexis Risk Solutions originally developed High Performance Computing Clus- 
ter (HPCC),° as a proprietary platform, for processing and analyzing large volumes 
of data on clusters of commodity servers more than a decade ago. It was turned into 
an open source system in 2011. Major components of an HPCC system include a 
Thor cluster and a Roxie cluster, although the latter is optional. Thor is called the 
data refinery cluster, which is responsible for extracting, transforming, and loading 


~http://hpcecsystems.com. 
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(ETL), as well as linking and indexing massive data from different sources. Roxie is 
called the query cluster, which is responsible for delivering data for online queries 
and online analytical processing (OLAP). 

Similar to Hadoop, HPCC also uses a distributed file system to support parallel 
processing on Big data. However, compared with HDFS, the distributed file system 
used by HPCC has some significant distinctions. First of all, HPCC uses two types of 
distributed file systems; one is called Thor DFS that is intended to support Big data 
ETL in the Thor cluster; the other is called Roxie DFS that is intended to support 
Big data online queries in the Roxie cluster. Unlike HDFS that is key-value pair 
based, the Thor DFS is record-oriented, which is flexible enough to support data 
sets of different formats, such as CSV, XML, fixed or variable length of records, and 
records with nested structures. Thor DFS distributes a file across all nodes in the 
Thor cluster with an even number of records for each node. The Roxie DFS uses 
distributed B+ tree for data indexing to support efficient delivery of data for user 
queries. 

HPCC uses a data-centric, declarative programming language called Enterprise 
Control Language (ECL) for both data refinery and query delivery. By using ECL, 
the user specifies what needs to be done on data instead of how to do it. The data 
transformation in ECL can be specified either locally or globally. Local transforma- 
tion is carried out on each file part stored in a node of the Thor cluster in a parallel 
manner, whereas global transformation processes the global data file across all nodes 
of the Thor cluster. Therefore, HPCC not only pioneers the current Big data com- 
puting paradigm that moves computing to where the data is, but also maintains the 
capability of processing data in a global scope. ECL programs can be extended with 
C++ libraries and compiled into optimized C++ code. A performance compari- 
son of HPCC with Hadoop shows that, on a test cluster with 400 processing nodes, 
HPCC is 3.95 faster than Hadoop on the Terabyte Sort benchmark test [41]. One 
of the authors of this chapter is currently conducting a more extensive performance 
comparison of HPCC and Hadoop on a variety of Big data analysis algorithms. More 
technical details on HPCC can be found in [24, 40, 41, 47]. 


4 Distributed Data Management Systems for Big Data 
Analytics 


As we discussed earlier in this chapter, huge volumes and a variety of data create a 
need for special types of data storage. In this section, we discuss recent advances in 
storage systems for Big data analytics and some commercially available cloud-based 
storage services. 
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4.1 Hadoop Distributed File System 


The Hadoop Distributed File System (HDFS)° is a distributed file system designed 
to run reliably and to scale on commodity hardware. HDFS achieves high fault- 
tolerance by dividing data into smaller chunks and replicating them across several 
nodes in a cluster. It can scale up to 200 PB in data, and 4500 machines in single 
cluster. HDFS is a side project of Hadoop and works closely with it. 

HDFS is designed to work efficiently in batch mode, rather than in interactive 
mode. Characteristics of typical applications developed for HDFS, such as write 
once and read multiple times, and simple and coherent data access, increases the 
throughput. HDFS is designed to handle large file sizes from Gigabytes to a few 
Terabytes. 

HDES follows the master/slave architecture with one NameNode and multiple 
DataNodes. NameNode is responsible for managing the file system’s meta data and 
handling requests from applications. DataNodes physically hold the data. Typically, 
every node in the cluster has one DataNode. Every file stored in HDFS is divided into 
blocks with default block size of 64 MB. For the sake of fault tolerance, every block 
is replicated into user-defined number of times (recommended to be a minimum of 3 
times) and distributed across different data nodes. All meta data about replication and 
distribution of the file are stored in the NameNode. Each DataNode sends a heartbeat 
signal to NameNode. If it fails to do so, the NameNode marks the DataNode as failed. 

HDFS maintains a Secondary NameNode, which is periodically updated with 
information from NameNode. In case of NameNode failure, HDFS restores a 
NameNode with information from the Secondary NameNode, which ensures fault- 
tolerance of the NameNode. HDFS has a built-in balancer feature, which ensures 
uniform data distribution across the cluster, and re-replication of missing blocks to 
maintain the correct number of replications. 


4.2 NoSQL Databases 


Conventionally, Relational Database Management Systems (RDBMS) are used to 
manage large datasets and handle tons of requests securely and reliably. Built-in 
features, such as data integrity, security, fault-tolerance, and ACID (atomicity, con- 
sistency, isolation, and durability) have made RDBMS a go-to data management 
technology for organizations and enterprises. In spite of RDBMS’ advantages, it is 
either not viable or is too expensive for applications that deal with Big data. This 
has made organizations to adopt a special type of database called “NoSQL” (Not 
an SQL), which means database systems that do not employ traditional “SQL” or 
adopt the constraints of the relational database model. NoSQL databases cannot pro- 
vide all strong built-in features of RDBMS. Instead, they are more focused on faster 
read/write access to support ever-growing data. 


Shttp://hadoop.apache.org. 
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According to December 2014 statistics from Facebook [16], it has 890 Million 
average daily active users sharing billions of messages and posts every day. In order 
to handle huge volumes and a variety of data, Facebook uses a Key-Value data- 
base system with memory cache technology that can handle billions of read/write 
requests. At any given point in time, it can efficiently store and access trillions of 
items. Such operations are very expensive in relational database management sys- 
tems. 

Scalability is another feature in NoSQL databases, attracting large number of 
organizations. NoSQL databases are able to distribute data among different nodes 
within a cluster or across different clusters. This helps to avoid capital expenditure 
on specialized systems, since clusters can be built with commodity computers. 

Unlike relational databases, NoSQL systems have not been standardized and fea- 
tures vary from one system to another. Many NoSQL databases trade-off ACID prop- 
erties in favor of high performance, scalability, and faster store and retrieve opera- 
tions. Enumerations of such NoSQL databases tend to vary, but they are typically 
categorized as Key-Value databases, Document databases, Wide Column databases, 
and Graph databases. Figure 2 shows a hierarchical view of NoSQL types, with two 
examples of each type. 


4.2.1 Key-Value Database 


As the name suggests, Key-Value databases store data as Key-Value pairs, which 
makes them schema-free systems. In most of Key-Value databases, the key is func- 
tionally generated by the system, while the value can be of any data type from a 
character to a large binary object. Keys are typically stored in hash tables by hashing 
each key to a unique index. 

All the keys are logically grouped, eventhough data values are not physically 
grouped. The logical group is referred to as a ‘bucket. Data can only be accessed 
with both a bucket and a key value because the unique index is hashed using the 
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Fig. 2 Categorization of NoSQL databases: The first level in the hierarchy is the categorization of 
NoSQL. Second level, provides examples for each NoSQL database type 
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bucket and key value. The indexing mechanism increases the performance of stor- 
ing, retrieving, and querying large datasets. 

There are more than 40 Key-Value systems available with either commercial or 
open source licenses. Amazon’s DynamoDB,’ which is a commercial data storage 
system, and open source systems like Memcached,® Riak,? and Redis!° are most pop- 
ular examples of Key-Value database systems available.These systems differ widely 
in functionality and performance. 

Key-Value databases are appropriate for applications that require one to store or 
cache unstructured data for frequent and long-term usages, such as chat applica- 
tions, and social networks. Key-Value databases can also be used in applications that 
require real-time responses that need to store and retrieve data using primary keys, 
and do not need complex queries. In consumer-faced web applications with high traf- 
fic, Key-Value systems can efficiently manage sessions, configurations, and personal 
preferences. 


4.2.2 Wide Column Database 


A column-based NoSQL database management system is an advancement over a 
Key-Value system and is referred to as a Wide Column or column-family database. 
Unlike the conventional row-centric relational systems [22], Wide Column databases 
are column centric. In row-centric RDBMS, different rows are physically stored in 
different places. In contrast, column-centric NoSQL databases store all correspond- 
ing data in continuous disk blocks, which speeds up column-centric operations, such 
as aggregation operations. Eventhough Wide Column is an advancement over Key- 
Value systems, it still uses Key-Value storage in a hierarchical pattern. 

In a Wide-Column NoSQL database, data are stored as name and value pairs, 
rather than as rows, which are known as columns. Logical grouping of columns is 
named as column-family. Usually the name of a column is a string, but the value 
can be of any data type and size (character or large binary file). Each column con- 
tains timestamp information along with a unique name and value. This timestamp 
is helpful to keep track of versions of that column. In a Wide-Column database, 
the schema can be changed at any time by simply adding new columns to column- 
families. All these flexibilities in the column-based NoSQL Systems are appropriate 
to store sparse, distributed, multidimensional, or heterogeneous data. A Wide Col- 
umn database is appropriate for highly scalable applications, which require built-in 
versioning and high-speed read/write operations. Apache Cassandra!! (Originated 
by Facebook) and Apache HBase”’ are the most widely used Wide Column data- 
bases. 


"http://aws.amazon.com/dynamodb/. 
Shttp://memcached.org/. 
*http://basho.com/riak/. 
http://redis.io/. 
'http://cassandra.apache.org/. 
!http://hbase.apache.org/. 
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4.2.3 Document Database 


A Document database works in a similar way as Wide Column databases, except that 
it has more complex and deeper nesting format. It also follows the Key-Value stor- 
age paradigm. However, every value is stored as a document in JSON,!? XML" or 
other commonly used formats. Unlike Wide Column databases, the structure of each 
record in a Document database can vary from other records. In Document databases, 
a new field can be added at anytime without worrying about the schema. Because 
data/value is stored as a document, it is easier to distribute and maintain data locality. 
One of the disadvantages of a Document database 1s that it needs to load a lot of data, 
even to update a single value in a record. Document databases have built-in approach 
of updating a document, while retaining all old versions of the document. Most Doc- 
ument database systems use secondary indexing [26] to index values and documents 
in order to obtain faster data access and to support query mechanisms. Some of the 
database systems offer full-text search libraries and services for real-time responses. 

One of the major functional advantages of document databases is the way it inter- 
faces with applications. Most of the document database systems use JavaScript (JS) 
as a native scripting language because it stores data in JS friendly JSON format. Fea- 
tures such as JS support, ability to access documents by unique URLs, and ability to 
organize and store unstructured data efficiently, make Document databases popular 
in web-based applications. Documents databases serve a wide range of web applica- 
tions, including blog engines, mobile web applications, chat applications, and social 
media clients. 

Couchbase!> and MongoDB! are among popular document-style databases. 
There are over 30 document databases. Most of these systems differ in the way data 
are distributed (both partition and replications), and in the way a client accesses the 
system. Some systems can even support transactions [23]. 


4.2.4 Graph Databases 


All NoSQL databases partition or distribute data in such a way that all the data are 
available in one place for any given operation. However, they fail to consider the rela- 
tionship between different items of information. Additionally, most of these systems 
are capable of performing only one-dimensional aggregation at a time. 


'Shttp://json.org. 

'hitp://www.w3.org/TR/2006/REC- xml 1 1-20060816/. 
'Shttp://couchbase.com/. 

l6http://mangodb.org/. 


Massive Data Analysis: Tasks, Tools, ... 25 


A Graph database is a special type of database that is ideal for storing and han- 
dling relationship between data. As the name implies Graph databases use a graph 
data model. The vertices of a graph represent entities in the data and the edges rep- 
resent relationships between entities. Graph data model, perfectly fits for scaling out 
and distributing across different nodes. Common analytical queries in Graph data- 
bases include finding the shortest path between two vertices, identifying clusters, 
and community detection. 

Social graphs, World Wide Web, and the Semantic Web are few well-known use 
cases for graph data models and Graph databases. In a social graph, entities like 
friends, followers, endorsements, messages, and responses are accommodated in 
a graph database, along with relationships between them. In addition to maintain- 
ing relationships, Graph databases make it easy to add new edges or remove exist- 
ing edges. Graph databases also support the exploration of time-evolving graphs by 
keeping track of changes in properties of edges and vertices using time stamping. 

There are over 30 graph database systems. Neo4j!” and Orient DB!®* are popu- 
lar examples of graph-based systems. Graph databases found their way into differ- 
ent domains, such as social media analysis (e.g., finding most influential people), 
e-commerce (e.g., developing recommendations system), and biomedicine (e.g., to 
analyze and predict interactions between proteins). Graph databases also serve in 
several industries, including airlines, freight companies, healthcare, retail, gaming, 
and oil and gas exploration. 


4.2.5 Cloud-Based NoSQL Database Services 


Amazon DynamoDB: DynamoDB’” is a reliable and fully managed NoSQL data 
service, which is a part of Amazon Web Services (AWS). It is a Key-Value database 
that provides a schema-free architecture to support ever-growing Big data in organi- 
zations and real-time web applications. DynamoDB is well optimized to handle huge 
volume of data with high efficiency and throughput. This system can scale and dis- 
tribute data, virtually, without any limit. DynamoDB partitions data using a hashing 
method and replicates data three times and distributes them among data centers in 
different regions in order to enable high availability and fault tolerance. DynamoDB 
automatically partitions and re-partitions data depending on data throughput and 
volume demands. DynamoDB is able to handle unpredictable workloads and high 
volume demands efficiently and automatically. 

DynamoDB offers eventual and strong consistency for read operations. Eventual 
consistency does not always guarantee that a data read is the latest written version 
of the data, but significantly increases the read throughput. Strong consistency guar- 
antees that values read are the latest values after all write operations. DynamoDB 
allows the user to specify a consistency level for every read operation. DynamoDB 


'http://neo4j.org/. 
'Shitp://www.orientechnologies.com/orientdb/. 
http://aws.amazon.com/dynamodb/. 
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also offers secondary indexing (i.e., local secondary and global secondary), along 
with the indexing of the primary key for faster retrieval. 

DynamoDB is a cost efficient and highly scalable NoSQL database service from 
Amazon. It offers benefits such as reduced administrative supervision, virtually 
unlimited data throughput, and the handling of all the workloads seamlessly. 

Google BigQuery: Google uses massively parallel query system called as 
‘Dremel’ to query very large datasets in seconds. According to [50], Dremel can 
scan 35 billion rows in ten seconds even without indexing. This is significantly more 
efficient than querying a Relational DBMS. For example, on Wikipedia dataset with 
314 million rows, Dremel took 10 seconds to execute regular expression query to 
find the number of articles in Wikipedia that include a numerical character in the title 
[50]. Google is using Dremel in web crawling, Android Market, Maps, and Books 
services. 

Google brought core features of this massive querying system to consumers as 
a cloud-based service called ‘BigQuery’.*? Third party consumers can access Big- 
Query through either a web-based user interface, command-line or through their own 
applications using the REST API. In order to use BigQuery features, data has to be 
transferred into the Google Cloud storage in JSON encoding. The BigQuery also 
returns results in JSON format. 

Along with an interactive and fast query system, Google cloud platform also pro- 
vides automatic data replication, on-demand scalability, and handles software and 
hardware failure without administrative burdens. In 2014, using BigQuery, scanning 
one terabyte of data only cost $5, with additional cost for storage.7! 

Windows Azure Tables: Windows Azure Tables?” is a NoSQL database tech- 
nology with a Key-Value store on the Windows Azure platform. Azure Tables also 
provides, virtually, unlimited storage of data. Azure Tables is highly scalable and 
supports automatic partitioning. This database system distributes data across multi- 
ple machines efficiently to provide high data throughput and to support higher work- 
loads. Azure Tables storage provides the user with options to select a Partition-Key 
and a Row-Key upfront, which may later be used for automatic data partitioning. 
Azure Tables follows only the strong consistency data model for reading data. Azure 
Tables replicates data three times among data centers in the same region and addi- 
tional three times in other regions to provide a high degree of fault-tolerance. 

Azure Tables is a storage service for applications with huge volume of data, and 
needs schema-free NoSQL databases. Azure Tables uses primary key alone and it 
does not support secondary indexes. Azure Tables provides the REST-based API to 
interact with its services. 


0nttp://cloud.google.com/bigquery/. 
*l https://cloud.google.com/bigquery/pricing. 
*2 http://azure.microsoft.com/. 
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5 Examples of Massive Data Applications 


In this section, a detailed discussion of solutions proposed by our research team for 
two real-world Big data problems are presented. 


5.1 Recommendations in e-Commerce 


Recommender systems are gaining wide popularity in e-commerce, as they are 
becoming major drivers of incremental business value and user satisfaction [29, 31]. 
In this section, we will describe the architecture behind a recommendation engine 
for eBay, a large open marketplace [28]. In an e-commerce system, there are two 
major kinds of recommendation scenarios: pre-purchase and post-purchase. 

In the pre-purchase scenario, the system recommends items that are good alter- 
natives for the item the user is viewing. In the post-purchase scenario, the recom- 
mendation system recommends items complementary or related to an item, which 
the user has bought recently. 


5.1.1 Architecture 


The architecture of the recommendation system, as illustrated in Fig. 3, consists of 
the Data Store, the Real-time Performance System, and the Offline Model Generation 
System. The Data Store holds the changes to website data as well as models learned. 
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Fig. 3 The recommendation system architecture with three major groups: The Offline Modeling 
System; The Data Store; The Real-time Performance System 
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The Real-time Performance System is responsible for recommending items using a 
session state of the user and contents from Data Store. The Offline Model Generation 
System is responsible for building models using computationally intensive offline 
analyze. Next, we present a detailed discussion about these components. 

Data Store: The Data Store provides data services to both the Offline Model Gen- 
eration and the Real-time Performance components. It provides customized versions 
of similar services to each of these components. For example, we consider a service 
that provides access to item inventory data. The Offline Modeling component has 
access to longitudinal information of items in the inventory, but not an efficient way 
of keyword search. On the other hand, the Real-time Performance System does not 
have access to longitudinal information, but it can efficiently search for item proper- 
ties in the current inventory. Two types of data sources are used by our system: Input 
Information sources and Output Cluster models. 


e Input Information Sources: 

The Data Store is designed to handle continuous data sources such as users’ actions 
and corresponding state changes of a website. At the same time, it also stores 
models, which are generated by the Offline Model Generation System. The data 
in the Data Store can be broadly categorized into inventory data, clickstream data, 
transaction data, and conceptual knowledge base. The inventory data contains the 
items and their properties. Clickstream data includes the raw data about the users’ 
actions with dynamic state of the website. Even though the purchasing history can 
be recreated from clickstream data, it is stored separately as transaction data for 
efficient access. Conceptual knowledge base includes ontology-based hierarchical 
organization of items, referred to as the category tree, lexical knowledge source, 
and term dictionary of category-wise important terms/phrases. 

e Output Cluster Model: The Data Store contains two types of knowledge struc- 

tures: Cluster Model and Related Cluster Model. The Cluster Model contains the 
definitions of clusters used to group the items that are conceptually similar. The 
clusters are represented as bag-of-phrases. Such a representation helps to cluster 
representatives as search queries and facilitates to calculate term similarity and 
item-coverage overlap between the clusters. 
The Related Cluster Model is used to recommend complementary items to users 
based on their recent purchases. This model is represented as sparse graph with 
clusters as nodes and edge between the clusters represents the likelihood of pur- 
chasing items from one cluster after purchasing an item in another cluster. Next, 
we discuss how these cluster models are used in the Realtime Performance System 
and, then, how they are generated using Offline Model Generation System. 


Real-time Performance System: The primary goal of the Real-time Performance 
System is to recommend related items and similar items to the user. It consists of two 
components, Similar Items Recommendation (SIR), which recommends users sim- 
ilar items based on current viewing item. Related Items Recommender (RIR) rec- 
ommends users the related items based on their recent purchases. Real-time Perfor- 
mance System is essential to generate the recommendations in real-time to honor the 
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dynamic user actions. To achieve this performance, any computationally intensive 
decision process is compiled to offline model. It is required to indexed data source 
such that it can be queried efficiently and to limit the computation after retrieving. 

The cluster assignment service generates normalized versions of a cluster as a 
Lucene” index. This service performs similar normalization on clusters and input 
item’s title and its static properties, to generate the best matching clusters. The SIR 
and RIR systems use the matching clusters differently. SIR selects the few best items 
from the matching clusters as its recommendations. However, RIR picks one item 
per query it has constructed to ensure the returned recommendations relates to the 
seeded item in a different way. 


Offline Model Generation: 


e Clusters Generation: The inventory size of an online marketplace ranges in the 
hundreds of millions of items and these items are transient, 1.e., covering a broad 
spectrum of categories. In order to cluster such a large scale and diverse inventory, 
the system uses distributed clustering approach on a Hadoop Map-Reduce cluster, 
instead of a global clustering approach. 

e Cluster-Cluster Relations Generation: An item-to-item co-purchase matrix is gen- 
erated using the purchase history of users from the transactional data set. Hadoop 
Map-Reduce clusters are employed to compute Cluster-related cluster pairs from 
the item-to-item co-purchase matrix. 


5.1.2 Experimental Results 


We conducted A/B tests to compare the performance of our Similar and Related 
Items Recommender systems described in this section over the legacy recommenda- 
tion system developed by Chen & Canny [11]. The legacy system clusters the items 
using generative clustering and later it uses a probabilistic model to learn relation- 
ship patterns from the transaction data. One of the main differences is the way these 
two recommendation system generate the clusters. The legacy system uses item data 
(auction title, description, price), whereas our system uses user queries to generate 
clusters. 

A test was conducted on Closed View Item Page (CVIP) in eBay to compare our 
Similar Items Recommender algorithm with the legacy algorithm. CVIP is a page 
that is used to engage a user by recommending similar items after an unsuccess- 
ful bid. We also conducted a test to compare our Related Items Recommender with 
legacy algorithm [11]. Both the test results show significant improvement in user 
engagement and site-wide business metrics with 90 % confidence. As we are not per- 
mitted to publish actual figures representing system performances, we are reporting 
relative statistics. Relative improvements in user engagement (Click Through Rate) 
with our SIR and RIR, over legacy algorithms, are 38.18 % and 10.5 %, respectively. 


*3http://lucene.apache.org/. 
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Fig. 4 Implementation flow diagram for supervised link prediction in biomedical literature 


5.2 Link Prediction in Biomedical Literature 


Predicting the likelihood of two nodes associating in the future, which do not have 
direct association between them in the current timestep, is known as the link predic- 
tion problem. Link prediction is widely used in social network analysis. Link predic- 
tion has wide range of applications such as identifying missing information, identify- 
ing spurious interactions, and studying the evolution of the network. In e-commerce, 
link prediction is used for building recommendation systems, and in bioinformatics, 
it is used to predict protein—protein interactions. 

Katukuri et al. [30], proposed a supervised link prediction method, to predict 
unknown association of medical concepts using biomedical publication information 
from Medline.** Medline is a National Institute of Health (NIH)’s citation database 
with more than 21 million publication citations. Figure 4 illustrates different stages 
in the proposed supervised link prediction approach. A temporal concept network is 
generated using relevant medical concepts extracted from publications. In the con- 
cept network, each node represents a medical concept and an edge between two nodes 
represents relationship that two medical concepts co-occurred at least in one publica- 
tion. Document frequency of a given concept is a weight of node and co-occurrence 
frequency of two concepts is edge weight. Now, link prediction problem is formu- 
lated as a process of identifying whether a pair of concepts, which are not directly 
connected in the current duration concept network, will be connected directly in the 
future. 

This link prediction problem is formulated as a supervised classification task. 
Training data is automatically labeled by comparing concept network snapshots of 
two consecutive time periods. This automatic labeling approach helps to avoid need 
for domain experts. 

In automatic labeling method, concept pairs, which are not directly connected in 
the first snapshot, are labeled based on its possible connection strength in the second 
snapshot. Connections strength is categorized as follows (S is edge weight in second 
snapshot, and minimum_support and margin (ranges between 0 to 1) are user-defined 
values): 


e Connection as strong: S > minimum_support 
e Connection as emerging: margin X minimum_support < S < minimum_support. 


**http://www.ncbi.nlm.nih.gov/pubmed/. 
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e Connection as weak: S < margin X minimum_support. 
e No connection: S=0. 


Given a pair of nodes that has no direct connection in first snapshot is assigned 
with positive class label if this pair is strongly connected in the second snapshot and 
is assigned negative class label if it has weak connection or no connection the second 
snapshot, and the pairs with intermediate values of strength are labeled as emerging. 

For each of labeled concept pairs, a set of topological features (random walk based 
and neighborhood-based) is extracted from the first snapshot of the concept net- 
work. The topological feature set also includes Common neighbors, Adamic/Adar, 
Jaccard Co-efficient, and Preferential Attachment [4]. Along with topological fea- 
tures, semantically enriched features, like Semantic CFEC [30] are extracted. Com- 
bining labeled concept pairs with the corresponding feature set generates the train- 
ing instances. Supervised classification algorithms, such as SVM, and C4.5 decision 
tree are used to generate prediction models. The classification accuracy of prediction 
models is calculated using cross-validation, which is on average 72 % [30]. 

Implementing such a computationally intensive phase to extract features needed 
for generating the predictive model on massive data needs large computational 
resources. For example, a snapshot of the concept network for years 1991—2010 has 
nearly 0.2 million nodes and nearly 44 million edges. Processing millions of publi- 
cations to extract medical concepts, generating class labels, and extracting features 
from large well-connected network is computationally heavy. To handle such com- 
putations on large graphs, the prediction system is developed by using the MapRe- 
duce framework and a Hadoop cluster environment. We implemented the MapRe- 
duce functions to extract the medical concepts from millions of publications in Med- 
line dataset, to generate the labeled data, and to extract structural features from the 
large concept graph. 

Implementing such a graph computation method on MapReduce framework has 
its own limitations. One of the drawbacks of MapReduce framework is its inability 
to retain the state of a graph across multiple iterations [44]. One approach to retain 
the state in a MapReduce pipeline is by explicitly writing the graph to disk after one 
iteration and reading it back from disk in the next iteration. This approach proves to 
be inefficient due to the huge increase in I/O operation and bandwidth [44]. 

Google proposed a Bulk Synchronous Parallel processing model called Pregel 
[37], which is a message passing model. Unlike MapReduce framework, this model 
helps by retaining the graph state across the iterations. Apache Giraph is open source 
alternative to Pregel, which is built on top of MapReduce framework. In such dis- 
tributed graph systems, a graph is partitioned and distributed among different cluster 
nodes. Each vertex has information about itself along with its neighbors. In our link 
prediction, features like the Jaccard Coefficient, can be extracted in parallel since 
such calculations depend only on information local to each vertex. However, other 
features, like the Semantic CFEC, need to be calculated by exploring all the paths 
between given pair of nodes, which can be formulated as an all-pairs-path problem. 
There are several frameworks [36, 37] that can calculate the all-pairs-path problem 
by passing information through edges, but between just a pair of nodes at a time. 
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However, these frameworks cannot support the operation of finding all paths between 
all pairs in parallel. In our case, there is a need to extract such features for millions 
of concept pairs. To the best of our knowledge, there is no algorithm or framework 
that can support such a problem to run in a distributed environment. This is one of 
the open computational challenges in graph analytics that needs to be investigated 
by the research community. 


6 Current Issues of Big Data Analytics 


In this section, we discuss several open challenges relating to computation, storage, 
and security in Big data analytics. 


6.1 Data Locality 


One prominent aspect in efficient Big data processing is the ability to access data 
without a significant latency. Given the transfer-prohibitive volume of Big data, 
accessing data with low latency can be accomplished only through data locality. In 
fact, data movement is possible in computations with moderate to medium volume 
of data where the data transfer to processing time ratio is low. However, this is not 
the case for Big data analytics applications. The alternative approach to alleviate the 
data transfer problem is moving the computation to where the data resides. Thus, 
efficient data management policies are required in the Big data analytics platforms 
to consider issues such as maximizing data locality and minimizing data migration 
(i.e., data transfer) between cloud data centers [63]. 

One of the key features of the Hadoop framework is its ability to take the effects 
of data locality into account. In Hadoop, the JobTracker component tries to allocate 
jobs to nodes where the data exists. Nonetheless, there are cases in which all the 
nodes that host a particular data node are overloaded. In this situation, JobTracker 
has to schedule the jobs on machines that do not have the data. 

To expedite data processing, Spark keeps the data in main memory, instead of 
on disk. Spark’s data locality policy is similar to Hadoop. However, in Spark, the 
Reduce tasks are allocated to machines where the largest Map outputs are generated. 
This reduces data movement across the cluster and improves the data locality further. 


6.2 Fault-Tolerance of Big Data Applications 


In long-running, Big data analytics applications, machine failure is inevitable. Both 
transient (1.e., fail-recovery) and permanent (1.e., fail-stop) failures can occur during 
the execution of such applications [32]. Google reports experiencing on average, five 
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machine crashes during a MapReduce job in March 2006 [12] and at a minimum 
one disk failure in each execution of a MapReduce job with 4000 tasks. Because of 
the criticality of failure, any resource allocation method for Big data jobs should be 
fault-tolerant. 

MapReduce was originally designed to be robust against faults that commonly 
happen at large-scale resource providers with many computers and devices such as 
network switches and routers. For instance, reports show that during the first year 
of a cluster operation at Google there were 1000 individual machine failures and 
thousands of hard-drive failures. 

MapReduce uses logs to tolerate faults. For this purpose, the output of Map and 
Reduce phases create logs on the disk [39]. In the event that a Map task fails, it is re- 
executed with the same partition of data. In case of failure in Reducer, the key/value 
pairs for that failed Reducer are regenerated. 


6.3 Replication in Big Data 


Big data applications either do not replicate the data or do it automatically through a 
distributed file system (DFS). Without replication, the failure of a server storing the 
data causes the re-execution of the affected tasks. Although the replication approach 
provides more fault-tolerance, it is not efficient due to network overhead and increas- 
ing the execution time of the job. 

Hadoop platform provides user a static replication option to determine the num- 
ber of times a data block should replicate within the cluster. Such a static replication 
approach adds significant storage overhead and slows down the job execution. A 
solution to handle this problem is the dynamic replication that regulates the replica- 
tion rate based on the usage rate of the data. Dynamic replication approaches help to 
utilize the storage and processing resources efficiently [64]. 

Cost-effective incremental replication [34] is a method, for cloud-based jobs, that 
is capable of predicting when a job needs replication. There are several other data 
replication schemes for Big data applications on clouds. In this section, we discuss 
four major replication schemes namely, Synchronous and Asynchronous replication, 
Rack-level replication, and Selective replication. These replication schemes can be 
applied at different stages of the data cycle. 

Synchronous data replication scheme (e.g., HDFS) ensures data consistency 
through blocking producer tasks in a job until replication finishes. Even though Syn- 
chronous data replication yields high consistency, it introduces latency, which affects 
the performance of producer tasks. In Asynchronous data replication scheme [32], a 
producer proceeds regardless of the completion of a producer task on a replica of the 
same block. Such a nonblocking nature of this scheme improves the performance 
of the producer. But consistency of Asynchronous replication is not as precise as 
the Synchronous replication. When Asynchronous replication is used in the Hadoop 
framework, Map and Reduce tasks can continue concurrently. 
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Rack-level data replication scheme ensures that all the data replicas occur on the 
same rack in a data center. In fact, in data centers, servers are structured in racks with 
a hierarchical topology. In a two-level architecture, the central switch can become 
the bottleneck as many rack switches share it. One instance of bandwidth bottleneck 
is in the Shuffling step of MapReduce. In this case, the central switch becomes over 
utilized, whereas rack-level switches are under-utilized. Using the Rack-level repli- 
cation helps reduce the traffic that goes through the central switch. However, this 
schema cannot tolerate rack-level failures. But recent studies suggest, the rack-level 
failures are uncommon, which justifies the adoption of Rack-level replication. 

In Selective data replication, intermediate data generated by the Big data appli- 
cation are replicated on the same server where they were generated. For example, in 
the case of Map phase failures in a chained MapReduce job, the affected Map task 
can be restarted directly, if the intermediate data from previous Reduce tasks were 
available on the same machine. The Selective replication scheme reduces the need 
for replication in the Map phase. However, it is not effective in Reduce phase, since 
the Reduce data are mostly consumed locally. 

Data replication on distributed file systems is costly due to disk I/O operations, 
network bandwidth, and serialization overhead. These overheads can potentially 
dominate the job execution time [65]. Pregel [37], a framework for iterative graph 
computation, stores the intermediate data in memory to reduce these overheads. 
Spark [66] framework uses a parallel data structures known as Resilient Distributed 
Datasets (RDDs) [65] to store intermediate data in memory and manipulate them 
using various operators. They also control the partitioning of the data to optimize 
the data placement. 


6.4 Big Data Security 


In spite of the advantages offered by Big data analytics on clouds and the idea of 
Analytics as a Service, there is an increasing concern over the confidentiality of 
the Big data in these environments [6]. This concern is more serious as increasing 
amount of confidential user data are migrated to the cloud for processing. Genome 
sequences, health information, and feeds from social networks are few instances of 
such data. 

A proven solution to the confidentiality concerns of sensitive data on cloud is to 
employ user-side cryptographic techniques for securing the data [6]. However, such 
techniques limit the cloud-based Big data analytics in several aspects. One limita- 
tion is that the cryptographic techniques usually are not transparent to end users. 
More importantly, these techniques restrict functionalities, such as searching and 
processing, that can be performed on the users’ data. Numerous research works are 
being undertaken to address these limitations and enable seamless Big encrypted 
data analytics on the cloud. However, all of these efforts are still in their infancy and 
not applicable to Big data scale processing. 
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Another approach to increase data confidentiality is to utilize multiple cloud stor- 
age units simultaneously [60]. In this approach, user data are sharded based on a user- 
side hashing function, and then the data for each cloud is encrypted and uploaded 
across multiple clouds. It is noteworthy that sharding and distributing of the data are 
achieved based on some lightweight user-side processing. Therefore, the control and 
distribution of the data is determined merely at the user-side. Although such shard- 
ing approach seems interesting for Big data analytics, challenges, such as processing 
sharded data across multiple clouds, still remains unsolved. 

Hybrid clouds have proven to be helpful in increasing the security of Big data 
analytics. In particular, they can be useful for cloud-based Big data analytics where 
a portion of the data is sensitive and needs specific trusted resources for execution. 
One approach is to label the data and treat them differently based on their labels [67]. 
As such, nonsensitive data are pushed to a public cloud for processing and sensitive 
data are processed in a private cloud. The coordination is accomplished through 
a scheduler placed within local resources that determines where a data should be 
processed depending on its label. 


6.5 Data Heterogeneity 


One of the major challenges researchers are facing is “How to integrate all the data 
from different sources to maximize the value of data.” In the World Wide Web 
(WWW), there is a huge amount of data created by social network sites, blogs, and 
websites. However, every source is different in data structure, semantics, and format. 
Structure of data from these sources varies from well-structured data (e.g., databases) 
to unstructured data (e.g., heterogeneous documents). 

Singh et al. [53], developed a framework to detect situations (such as epidemics, 
traffic jams) by combining information from different streams like Twitter, Google 
Insights, and satellite imagery. In this framework, heterogeneous real-time data 
streams are combined by converting selected attributes and unified across all streams. 
There are several other proposed frameworks that can combine different data sources 
for various chosen domain-specific applications. Most of these solutions use a 
semantics-based approach. Ontology matching is a popular semantics-based method, 
which finds the similarity between the ontologies of different sources. 

Ontology is a vocabulary and description of a concept and its relationship with 
others in the respective domain. In the Web example, ontology is used to trans- 
form unstructured or partially structured data from different sources. Most of them 
are human readable format (e.g., HTML) and are hard for the machine to under- 
stand. One of the most successful ontology integration projects is Wikipedia, which 
is essentially integrated with human intervention. Semantic Web tools are used to 
convert the unstructured Web to a machine understandable structure. Semantic Web 
adds ontology constructs into web pages and enables machines to understand the 
contents of a webpage. It helps to automate integration of heterogeneous data from 
different sources on the Web using ontology matching. 


36 M.K. Pusala et al. 


7 Summary and Discussion 


Size of data in the digital universe is almost doubling every year and has reached to 
a stage that they cannot be processed by the conventional programming, computing, 
storage, visualization, and analytical tools. In this study, we reviewed different types 
of analytic needs arising in research and industry. We broadly categorized the cur- 
rent analytical applications as descriptive, predictive, and prescriptive and identified 
several real-world applications of each type.Then, we provided an overview of the 
state-of-the-art on platforms, tools, and use cases for massive data analytics. 

In particular, we discussed that cloud services are helpful platforms in alleviat- 
ing many of the massive data processing challenges. MapReduce compute resources, 
NoSQL databases, virtually unlimited storages, and customized filesystems, amongst 
many others, are useful cloud services for massive data analytics. 

We provided two use cases that were investigated within our research team. The 
first one, recommendation in e-commerce applications, consists of anumber of com- 
ponents that can be partitioned into three major groups: The Data Store that contains 
data about the active and temporarily changing state of an e-commerce web site; The 
Real-time Performance System that generates recommendations in real-time based 
on the information in the Data Store; and the offline model generation that conducts 
computationally intensive offline analyses. The Real-time Performance System con- 
sists of two components, similar items recommender (SIR) and related items recom- 
mender (RIR). Both of these components take a seed item as input, and return a set 
of items that are similar or related to that seed item. 

The second use case addresses the problem of link prediction that proposes associ- 
ations between medical concepts that did not exist in earlier published works. In this 
project, we model biomedical literature as a concept network, where each node rep- 
resents a biomedical concept that belongs to a certain semantic type, and each edge 
represents a relationship between two concepts. Each edge is attached with a weight 
that reflects the significance of the edge. Based on the constructed massive graph, a 
machine-learning engine is deployed to predict the possible connection between two 
indirectly connected concepts. 

In the course of our research on massive data analytics tools and projects, we 
have learnt key lessons and identified open challenges that have to be addressed by 
researchers to further advance efficient massive data analytics. Below, we highlight 
some of the lessons and challenges: 


e In many analytical applications (e.g., recommendation system in e-commerce), 
even with availability of state-of-the-art Big data technologies, treating customer 
data as a data stream is not yet viable. Therefore, some steps (e.g., model building 
in recommendation systems), have to be performed offline. 

e It is difficult, if not impossible, to come up with a generic framework for various 
types of analytics. For instance, in the recommendation system, which is an exam- 
ple of predictive data analytics in e-commerce, there are many subtle nuances. 
Thus, a specific architecture is required based on the merits of each application. 
Accordingly, in the eBay application, we noticed that Related Items Recommen- 
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dation (RIR) needs a different architecture compared to Similar Items Recommen- 
dation (SIR). 

Many Big data analytics (e.g., biomedical link prediction) process massive graphs 
as their underlying structure. Distributed graph techniques need to be in place 
for efficient and timely processing of such structures. However, to the best our 
knowledge, there is not yet a comprehensive distributed graph analytic framework 
that can support all conventional graph operations (e.g., path-based processing in 
distributed graphs). 

Data locality and replication management policies ought to be cleverly integrated 
to provide robust and fault-tolerant massive data analytics. 

As massive data are generally produced from a great variety of sources, novel, 
semantics-based solutions should be developed to efficiently support data hetero- 
geneity. 
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Statistical Challenges with Big Data 
in Management Science 


Arnab Laha 


Abstract In the past few years, there has been an increasing awareness that the enor- 
mous amount of data being captured by both public and private organisations can 
be profitably used for decision making. Aided by low-cost computer hardware, fast 
processing speeds and advancements in data storage technologies, Big Data Ana- 
lytics has emerged as a fast growing field. However, the statistical challenges that 
are faced by statisticians and data scientists, while doing analytics with Big Data 
has not been adequately discussed. In this paper, we discuss the several statistical 
challenges that are encountered while analyzing Big data for management decision 
making. These challenges give statisticians significant opportunities for developing 
new statistical methods. Two methods—Symbolic Data Analysis and Approximate 
Stream Regression—which holds promise in addressing some of the challenges with 
Big Data are discussed briefly with real life examples. Two case studies of applica- 
tions of analytics in management—one in marketing management and the other in 
human resource management—are discussed. 


1 Introduction 


If we look at the evolution of the adoption of analytics in businesses over time we 
see that there are several distinct stages. It all began in the early 1980s with ‘standard 
reports’ that described what had happened to a business function within a specified 
time period. These reports could be information about how many transactions took 
place in a branch, how much quantity of a product was sold, what was the amount 
spent on a certain stock keeping unit (SKU) by the loyalty card holders of a retailer, 
etc. The reports had to be preprogrammed and were produced at specified times such 
as quarterly, half-yearly or annually. With the advent of relational database systems 
(RDBMS) and the structured query language (SQL) ad-hoc reporting became pos- 
sible. The managers could query their organisation’s databases to seek information 
and reports which were not preprogrammed and were not routinely required. The 


A. Laha (œx) 
Indian Institute of Management Ahmedabad, Ahmedabad, India 
e-mail: arnab @iima.ac.in 


© Springer India 2016 41 
S. Pyne et al. (eds.), Big Data Analytics, DOI 10.1007/978-81-322-3628-3_3 


42 A. Laha 


ability to query the database and drill-down as and when required allowed managers 
to figure out root causes of problems and helped them in eliminating these. It also 
enabled creation of alerts that were triggered to draw attention of managers for taking 
appropriate action when necessary. To give an example of such an alerting system, 
think of a bank customer approaching a teller with a request for encashing a cheque. 
The system would automatically alert the concerned bank manager for clearance of 
the cheque by sending a message on his desktop. Once the manager cleared the check 
the system would then alert the teller that the approval has been obtained through a 
message on the teller’s desktop who can then make the payment to the customer. 

In this context, it may be noted that much of the development of statistics in the 
early years of the twentieth century was driven by applied problems coming from 
different areas such as manufacturing, agriculture, etc. Two prominent examples 
being the work of Walter Shewhart who developed statistical process control (SPC) 
at the Western Electric’s Hawthorne plant and the work of Ronald A. Fisher, who 
in the context of agricultural field experiments at Rothamstead Experimental Sta- 
tion, developed the field of design of experiments (DoE). Both of these techniques 
continue to be used by industries across the world for improving their quality and 
productivity. Seeing the great potential of using statistics in improving the quality of 
life in an emerging country like India, Mahalonobis [13] described statistics as a key 
technology and advocated its wider adoption in formulation of public policy and in 
solving challenging problems facing the country. 

In the later years of the 1990s, statistical models started to be used much more 
widely than before in organisations to help them gain competitive advantage over 
their rival organisations. Since the early adopters reported substantial gains because 
of the use of analytical models, adoption of analytics for business decision making 
accelerated and spread to a variety of industries such as consumer products, indus- 
trial products, telecommunications, financial services, pharmaceuticals, transport, 
hospitality and entertainment, retail, e-commerce, etc. ([6] p. 7). These statistical 
models primarily helped managers by giving them insights (why a certain phenom- 
ena happened and what would happen if some things were done differently), fore- 
casting (demand for different SKUs, sales, profit, etc.) and predicting occurrence or 
non-occurrence of some events (like attrition of customers, defaults on loans, etc.). 
Some organisations like American Airlines could use the vast data available with 
them to increase their revenue by adopting practices such as dynamic pricing and 
overbooking using statistical demand forecasting and optimization techniques [23]. 
Hewlett-Packard could ensure that on its online sales portal HPDirect.com, the right 
customer received the right offer at the right time leading to higher conversion rates 
and order sizes [26]. 

The use of statistical techniques on large databases for business decision making 1s 
now a common phenomenon in the developed world and is fast catching up in emerg- 
ing economies. The data gathering abilities has increased tremendously over the last 
decade. With data coming from a variety of sources for example posts to social media 
sites like Facebook or Twitter, digital pictures and video uploaded by users, stream- 
ing video from surveillance cameras, sensors attached to different devices, mobile 
phone records, purchase transaction records, searches conducted on websites such as 
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Google the volume of data potentially available for decision making is truly huge. It 
is commonly estimated that more than 90 % of the total data generated by mankind 
has been generated in the last two years. It is therefore not surprising that the sta- 
tistical methods developed in an era where even a sample of size 30 was considered 
large would face challenges when applied to extract information from such very large 
databases. As we shall see later in this article volume of data is not the only challenge 
that is faced by statisticians dealing with ‘Big Data’. In fact with generous support 
coming from scientists working in the field of computer science, computational chal- 
lenges can hopefully be overcome much sooner than the other challenges. Since with 
every challenge comes an opportunity, “Big Data’ offers statisticians opportunities 
for developing new methods appropriate for dealing with these kind of data. 


2 Big Data 


One of the key characteristics of Big Data used for business decision making is that 
it is often collected for a purpose, which is different from the purpose for which it 
is being analysed. In fact, Landefield ({11], p. 3) defines Big Data as the ’use of 
large-scale business and administrative data sets that are being used for secondary 
purposes other than those for which the data was ordinarily collected’. It is this aspect 
of Big Data that makes proper statistical analysis and decision making using Big Data 
particularly challenging. 

Big Data is usually characterised by 4Vs—Volume, Velocity, Variety and Verac- 
ity. The total Volume of data is generally very high—several terabytes or more, the 
data acquisition rate (Velocity) is generally very high, different types of data (such 
as numerical, categorical, ordinal, directional, functional, text, audio, images, spatial 
locations, graphs) may be present in the same database (Variety) and these different 
types of data may be related with one another, and finally poor quality (Veracity) of 
the data because it is commonly seen that the data has been collected over a period 
of time by different agencies for different purposes. The final V is particularly chal- 
lenging as it is difficult to clean such high volumes of data with more data coming 
in from multiple sources. 

The key enablers in the present growth of Big Data are rapid increase in stor- 
age capabilities making storage inexpensive, rapid increase in processing power and 
developments in distributed storage mechanisms, availability of huge volume of data 
from diverse sources, and last but not least, a better value for money proposition for 
businesses. To understand why Big Data is a better value for money proposition for 
businesses, let us consider a few examples below. 

Everyday, a large volume of messages are uploaded on social networking sites 
such as Facebook, Twitter, etc. These messages provide a storehouse of opinions 
and sentiments about a variety of products, services, regulatory policies, effective- 
ness of governance, etc. Sentiment analysis aims at analysing such vast amount of 
messages using techniques of text analytics to find out the opinion of the target cus- 
tomer segment of a product or service. These opinions or sentiments can be classified 
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as positive, negative or neutral and managerial action can be initiated on this basis. A 
change in sentiment of the customers in the target segment of a product (or service) 
from positive to negative needs to be immediately addressed to because it presents an 
opportunity to competitors to wean away the customers. Hence, most large compa- 
nies operating in consumer durables or fast moving consumer goods (FMCG) indus- 
try actively monitor the sentiment of their customers and aim to maintain a high 
positive rating. You may note that all the 4Vs pose a challenge to analysis of such 
data. 

Another example of use of analytics is in predicting customer churn in the highly 
competitive telecom industry. Since customer acquisition cost in this industry is quite 
large companies aim to retain their most valuable customers through a variety of 
means such as giving attractive service offers, discounts, freebies, etc. For almost all 
companies in this industry, a large number of their customers are occasional users of 
their services and do not generate high revenues whereas only a few of them generate 
the bulk of their revenues. Early identification of churn intentions of any member of 
this group of high revenue generators helps the company in taking appropriate steps 
to prevent the churn. Analysis of the very large volume of call records often throws 
up interesting patterns which enables these companies to provide customised offers 
of value to these customers thereby retaining them. Any social media message carry- 
ing negative sentiment about the service or positive sentiment about a competitor’ s 
service from a member of this select customer group can often be seen as a lead- 
ing indicator about their churn intention. Thus, one needs to combine information 
of one’s own database, here for example the call record details, with the information 
about the same customer’s behaviour on the social networking sites, to develop a 
more holistic view of the customer’s level of satisfaction with the service and gauge 
the extent of his churn intention. Again, in this case also all the 4Vs pose challenge 
to analysis of this data. 

As a third example, consider the problem of stopping credit card frauds. Analytics 
play an important role in inferring from the customers past behaviour whether the 
present transaction is genuine or is a possible case of fraud which needs to be stopped. 
Using transaction related information available with the credit card company such 
as the amount, the place where the card is swiped, time, merchant history, etc., and 
comparing the same with the customer’s past transaction records a probability of 
fraud for the current transaction can be calculated. If this probability turns out to be 
large the transaction is denied and a message is sent to the customer asking him/ her 
the validity of the attempted transaction or seeking identification details. If the sought 
information is successfully provided then the transaction is subsequently allowed 
else a note is taken about this fraud attempt and the company may replace the credit 
card of the customer immediately with a new card. The entire exercise of determining 
whether an alert should be raised or not has to be done within a few seconds making 
this apparently straightforward binary classification task challenging in the context 
of Big Data. 

As the fourth and final example, let us consider the huge stream of videos that are 
available from the surveillance cameras of a large facility like an airport. The aim 
of surveillance at the airport is mainly to prevent happening of untoward incidents 
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by identifying suspicious movements, unclaimed objects and presence of persons 
with known criminal records. The task here is humongous akin to finding a needle 
in a haystack. Video analytics can be used to analyse video footages in real time 
for detecting the presence of suspicious persons or unclaimed objects amongst the 
thousands of genuine travellers. In this case the volume, velocity and variety of data 
pose a challenge to real-time analysis of this data. 


3 Statistical Challenges 


In this section, we discuss different challenges that are encountered while analyzing 
big data from a statistical perspective. We structure our discussion around the 4Vs 
mentioned in the earlier section. 


3.1 Volume 


As we have discussed earlier, advancement in data acquisition and storage technol- 
ogy has made it possible to capture and store very large amounts of data cheaply. 
However, even with very high speed processors available today some of the simple 
statistical methods which involve sorting (such as the computations of quantiles) or 
matrix multiplication (such as linear regression modelling) face computational chal- 
lenges when dealing with Big Data. This is so because the popular algorithms for 
sorting or matrix multiplication do not scale-up well and run very slow on terabyte- 
scale data sets. For example, the very popular sorting algorithm, Bubble Sort is not 
suitable for use with large data sets since its average and worst case time complexity 
are both O(n”) where n is the number of data elements. Instead, it would be better to 
use an algorithm such as Heapsort whose average and worst case time complexity 
are both O(n log n) or the Radix Sort whose average and worst case time complexity 
are both O(nk) where k in general is a function of n [5] (see Cormen et al. 2009 for 
more details regarding sorting algorithms and their computational complexity). 
Matrix multiplication is ubiquitous in statistics right from the theory of linear 
models to modern advanced methods such as Nonparametric Regression. However, 
the computational complexity of matrix multiplication is high since for computing 
C nxn = AmxkPrxyn USING the standard method we need O(mkn) computations. In the 
special case when m = k = n, this reduces to O(n?) computations. Thus in the full- 
rank Gauss—Markov model Y„xı = X,x¢Bixt + Ey, the computation of the BLUE 


of P which is Bret = (XTX i: XTY becomes very time consuming for large n (see 
Rao, 1973 for details of the Gauss—Markov model and its applications). Hence, faster 
algorithms such as the Strassen’s algorithm [24] or the Coppersmith—Winograd algo- 
rithm [4] need to be used for carrying out such computations with very large data. 
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3.2 Velocity 


Some of the fundamental ideas of statistics come under challenge when dealing with 
Big Data. While ‘optimality of statistical procedures’ has been the ruling paradigm 
in Statistics for decades, the speed of computation has never been of primary concern 
to statisticians while developing statistical methods until very recently. In manage- 
ment applications, it is often expected that the acquired data would be analysed in 
the shortest possible time so that necessary action can be taken as quickly as pos- 
sible. For example, in real-time surveillance for financial fraud, it is important to 
conclude whether a proposed transaction is a fraudulent one or not in a matter of few 
seconds. Analysis of such “streaming data’ opens up new challenges for statisticians 
as it is not possible to come to a decision after analyzing the entire dataset because 
of severe time constraints. A general feature of analysis of streaming data is that a 
decision regarding whether there is some change in the data generating mechanism 
is sought to be arrived at by looking only at a very small part of the available data. 
This calls for a paradigm shift regarding the choice of statistical procedures when 
dealing with streaming data. Instead of choosing statistical procedures based on the 
present paradigm of ‘Get the best answer’ we need to choose procedures based on 
the paradigm ‘Get a good answer fast . 

Streaming data is not entirely new to the statistical community. In online process 
control, it is assumed that the data is streaming but possibly at a much slower rate 
than in the Big Data applications. The widely used statistical process control (SPC) 
tools such as X-chart, CUSUM chart, EWMA charts, etc., have been devised keeping 
in mind the streaming nature of data (see [16] for details regarding control charts). 
Thus, these techniques with suitable adaptations and modifications have potential to 
be applied with streaming data even in the Big Data context. One of the key differ- 
ences of the streaming data in the Big Data context with that in the manufacturing 
context is that in the latter context a probabilistic structure for the streaming data can 
be assumed which may not even exist in the former context and even if it does it may 
be highly non-stationary [27]. While designing algorithms for working with data 
streams, it is important to remember that data cannot be permanently stored. This 
implies that the algorithms will have to discard the data after using them. Another 
requirement is that the results should be swiftly updated with current data getting 
more weight than historical data. To understand this point, let us suppose our inter- 
est is in getting an estimate of the current mean of the data stream. The computation 
of arithmetic mean of a data stream can be done simply by recursively computing 


but this responds very slowly to any change in the process mean as it gives very 
low weight to the current data compared to historical data. An alternative is to use 
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the exponentially weighted moving average (EWMA) Y,, which can be computed 
recursively as 
Y, =OY,_,+U-4@X,,0<@< 1. 


It is easy to see that Y, responds to any change in mean of the data stream much 
faster than X,,, while giving the correct mean when there is no change. 

Often, Big Data does not come from a single population but is a mixture of data 
coming from many populations. This problem is not entirely new and procedures 
for dealing with mixture of populations, such as cluster analysis, has been in exis- 
tence for quite some time. Standard statistical procedures such as linear regression 
analysis does not perform well when dealing with data which is a mixture of dif- 
ferent populations. A naive approach is to first cluster the data using a clustering 
algorithm such as K-means algorithm and then carry out linear regression in each 
of these clusters separately treating each cluster as a ‘population’. Desarbo and Cron 
[7] suggested a method for doing clusterwise linear regression using a maximum 
likelihood approach using the mixture of Gaussian distributions setup. In this case 
the regression parameters in each cluster are simultaneously estimated. Alternative 
nonparametric approaches include procedures such as classification and regression 
trees popularly known as CART [3]. 

The additional challenge that streaming data poses is that the number of popula- 
tions (clusters) may not be known in advance and that it may vary over time. This 
is often seen in social media data where new groups with distinct characteristics 
suddenly emerge and sometimes established groups decay and eventually disappear. 
Thus finding estimates, trends or patterns can become a very difficult exercise intro- 
ducing unique computational and statistical challenges. Silva et al. [22] gives a com- 
prehensive survey of the work done in the area of clustering data streams data in 
recent years. 


3.3 Variety 


Big data is often a mixture of different types of data leading to the problem of Vari- 
ety. The well-known statistical methods either deal with numerical or categorical 
data. Thus when dealing with mixture of data types such as video data and categor- 
ical data, the current statistical methods are not of much use. To give an example 
of how such data may arise in practise consider the following situation: A video is 
requested by an user from a website such as Youtube and suppose an advertisement 
video precedes the requested video. The user may watch the advertisement video in 
full or watch it partially or may entirely skip it. A creative content company may 
be interested in knowing the characteristics of the advertisement video that would 
induce the user to see it in full or with what kind of videos the advertisement video 
should be bundled so that the chance the user sees it to the full is maximised. Situ- 
ations where multiple types of data occur together are also quite common in many 
areas of science such as speed and direction of wind in meteorology (linear data and 
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angular data), variation in shape of skull across humans of different ethnicity (shape 
data and categorical data), variation of blood pressure and development of heart 
disease (functional data and categorical data), etc. Over the last two decades, our 
knowledge of analyzing non-numerical non-categorical data has increased substan- 
tially. Mardia and Jupp [15] give a book length treatment of analysis of directional 
data with particular emphasis on analyzing circular and spherical data (see also [9]). 
A detailed account of the analysis of shape data can be found in Dryden and Mar- 
dia [8]. Bhattacharya and Bhattacharya [1] discusses nonparametric inference on 
manifolds. Ramsey and Silverman [19] give an excellent introduction to functional 
data analysis. More such new methodological developments are needed for drawing 
sound inference with data of different types. The field of Symbolic Data Analysis 
holds much promise in addressing the Variety problem encountered with Big Data 
as it deals with different data types together (see Billard [2] and Noirhomme-Fraiture 
and Brito [18] for more details). 


3.4 Veracity 


Veracity of data is a big concern when dealing with Big Data. The data can be inaccu- 
rate because of many possible reasons including intentional falsification. In reviews 
of hotels, films, cars, etc. it is difficult to decide whether the review has been given 
in a fair manner or is motivated by reasons other than the merit of the product or 
service. Though it is difficult to weed out false reviews completely many websites 
are attempting to develop checks which identifies and weeds out false reviews. Also 
mechanisms are being evolved to deter people from writing false reviews. However, 
at present there are hardly any statistical methods available that can detect and weed 
out false data from a given data set automatically. Thus, it is presently a herculean 
task to clean up such large volumes of data before doing any analysis. Better auto- 
mated methods for identifying and eliminating incorrect data are an urgent require- 
ment. 

Karr et al. [10] defines Data Quality as ‘the capability of data to be used effec- 
tively, economically and rapidly to inform and evaluate decisions’. Since the Big Data 
are often collected for a purpose different from that for which it is being analysed, 
there is no sampling design and it is well known to statisticians that such data may 
induce bias which may lead to conclusions which are not generalizable. Moreover, 
there is possibility that the data definitions as well as the mechanism of data collec- 
tion may have changed over time. For example in credit default prediction studies 
the persons seeking loan from an institution may change over a period of time based 
on bank’s behaviour in terms of giving loans. Hence, Data Quality is often poor in 
Big Data. 

Most of the currently known statistical methods rely on the assumption that 
the data being analysed is a random sample from the underlying population. This 
assumption is generally not valid when working with Big Data in the field of Man- 
agement. In almost all encounters with Big Data in Management it is seen that the 
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data is non-random. Therefore, it is of great importance to validate any insight that is 
obtained through analysis of Big Data, using data which has been collected using a 
statistically sound method before taking any important decision based on that insight. 


3.5 Privacy and Confidentiality 


Last but not the least is the concern that Big Data analytics can threaten privacy and 
confidentiality and can impinge on an individual’s personal freedom. In recent years 
we have seen several instances where an organisation or a state agency has attempted 
to predict the status of an individual for reasons ranging from purely commercial to 
involving national security using Big Data. Such use of Big Data raises substantial 
questions of ethics and calls for new frameworks and guidelines to prevent misuse 
of such data. 

In many situations, particularly when organisations have to release data for public 
use, it becomes important to treat the data in a manner that prevents recovery of indi- 
vidual level confidential information from the released data. Reiter [20] discusses in 
detail the statistical approaches to protecting data confidentiality that are commonly 
used and the impact that it has on the accuracy of secondary data analysis. Since 
most data that are currently generated is being stored with the expectation that its 
analysis in future, in possibly different context, may help in generating new insights 
for the organisation, it is extremely important that such data be treated for protecting 
data confidentiality before storage. 


4 Statistical Methods for Big Data 


In the earlier section, we have discussed the various statistical challenges that are 
brought forth by Big Data. In this section, we briefly discuss two statistical methods 
namely Symbolic Data Analysis and Approximate Stream Regression which can be 
used to tackle the Variety and Velocity problems of Big Data, respectively, in some 
applications. 


4.1 Symbolic Data Analysis 


Symbolic data arise in Big Data context through aggregation of data for certain fea- 
tures of interest. For example, a credit card company need not store the entire his- 
tory of credit card transactions of its customers but instead may store only the min- 
imum and maximum amounts transacted by them. Thus for each customer the bank 
stores an interval, which is the range of the amounts transacted by the customer. 
Such data are refereed to as interval-valued symbolic data. Another possibility is 
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that it creates a histogram (or frequency distribution) of the transaction amounts and 
stores the same. In this case the data is histogram-valued. There can be many other 
forms of symbolic data as discussed in Billard [2]. In the following we restrict our- 
selves to interval-valued symbolic data. Let [a;, b;], i = 1,...,n be a random sample 
of n intervals. Assuming that the variable of interest (say, credit card transaction 
amounts) of the i-th customer are uniformly distributed in the interval [a,, b;] we can 
compute the mean O}; = &tb: and the variance 0,,= 4)" The random variables 
©, = (©,,, ©,;) are referred to as the internal parameters of the i-th interval [a,, b;]. 
Le-Rademacher and Billard [12] assumes that the random variables ©}; and @,, are 
independent with ©,, distributed as N(y, o°) and ©,, distributed as Exp(f) for all 
i=1,...,n. They obtain the MLEs of n,o and p based on the observed interval- 
valued data. As an illustration, we apply this technique to examine the temperature 
of India in the months of October-December. Historical data of the maximum and 
minimum temperatures during these months for the years 1901—2011 are available 


from the weblink 


https://data.gov.in/catalogs/ministry department /india- 


meteorological-department-imd 


We compute the internal parameters ©}; and ©,, for the given data using the for- 
mula discussed above. We find that it is reasonable to assume both ©}; and ©}; 
to be normally distributed using the Ryan-Joiner normality test implemented in 
Minitab 16 software. We assume ©}, to be distributed as N(u,07) and @,, to be 
distributed as N(v, t7). The MLEs of the parameters are found to be fi = 21.82,6 = 
0.44, 0 = 9.44 and t = 0.91. Now note that if X is distributed as Uniform(a, b) then 


a = E(X) — /3SD(X) and b = E(X) + \/3SD(X). Hence, we may consider the inter- 


val [21.82 — y3 x 9.44, 21.82 + y3 x 9.44] = [16.50, 27.14] as the average range 
of temperature during the months of October-December in India. 


4.2 Approximate Stream Regression 


In many applications such as predicting the amount of monthly spend by a customer 
on a certain category of products, multiple regression models are used. As the data 
accumulates over time, often a phenomenon referred to as concept drift is encoun- 
tered. In simple words, this means that the regression coefficients may change over a 
period of time. This shows up in larger prediction errors leading to rebuilding of the 
regression model with current data. However, it is preferable to have a method which 
adapts to the concept drift, i.e. the regression coefficients are automatically updated 
to its current values without needing to rebuild the model which is often a costly exer- 
cise. Nadungodage et al. [17] proposes the Approximate Stream Regression (ASR) 
to address this problem. One of the key concerns with high velocity streaming data is 
that the entire data cannot be stored. Thus the data which has been used has to be dis- 
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carded and cannot be used again in future. In ASR the data is accumulated in blocks, 
say of size m. Let Jý be the vector of regression coefficients computed using the data 
of the k-th block only and let £,_, be the vector of regression coefficients computed 
at the end of processing of the first k — 1 blocks, then J, is computed according to 
the formula, 


P= 0- a) pr + apr 


where 0 < a < 1. It is easy to see that 
B, = (1 — a) BF +a — a) Be +- apt +... +07 (1 — a) BS + a" B 


The value of a determines the emphasis that is given on current data. A value of 
a close to O indicates that the current data is emphasised more, whereas a value 
of a near | indicates that the historical data is given more importance. We now 
illustrate using an example the advantage of using ASR over the common approach 
of building a regression model and then using it for prediction in the context of 
streaming data with concept drift. For this purpose we generate six blocks of data 
with each block having 30 observations on Y, X, and X,. The variable Y is taken as 
the response variable and the variables X, and X, are taken as predictor variables. The 
variables X, and X, are generated from normal distribution with mean 0 and standard 
deviation 1. In the first three blocks we compute Y as Y = X, + X, + €, whereas in 
the remaining three blocks we compute Y as Y = 1 + 2X, — X, + e where the errors 
(e) are generated from normal distribution with mean 0 and standard deviation 0.5. 
All computations are done using Minitab 16 software. The value of a for ASR is 
fixed to be 0.75. The ASR method is compared with the linear regression model 
built by using only the data for Block 1. For this method the predicted values in 
Blocks 2—6 are obtained using the regression coefficients estimated on the basis of 
Block | data. The two methods are compared on the basis on Mean Square Predictive 
Error (MSPE). The estimated regression coefficients in each block, the regression 
coefficients obtained using the ASR method and the MSPE of the ASR method and 
that of the fixed coefficients method are given in Table 1. 


Table 1 Comparison of predictive performance of ASR with linear regression for streaming data 
with ai drift (a = 2 ; 


ASR Fixed (ASR [ASR ASR 


OALA, AION =e 
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It is seen that the ASR method performs much better than the fixed coefficients 
method in terms of having lower MSPE after the information of the changed regres- 
sion coefficients are fed into the ASR model. Further it is seen that the regression 
coefficients of the ASR method steadily move towards the new values as more infor- 
mation become available. Thus it is expected that ASR would prove to be a useful 
method for doing regression with streaming data. 


5 Case Studies 


In this section, we look at two case studies of application of Big Data in management. 
The first case study looks at the problem of cross selling in an online environment 
without any human intervention and suggest a methodology for the same and the sec- 
ond case study discusses the improvements achieved in the talent acquisition process 
of a large Indian IT company using innovative analytical solutions. 


5.1 Online Recommendations for Cross Selling 


In today’s competitive environment, cost of acquiring a customer is ever rising. 
Hence, organisations are attempting to realise better value from their existing cus- 
tomers by selling other useful products (or services) to them. For this purpose, it 
is important for an organisation to identify the products (or services) that would be 
most beneficial to each individual customer and to provide him/her with the best 
value proposition for the same. Many organisations use analytics to find out the best 
offer for each customer and then use human customer service representatives to com- 
municate the same to him/her. However this is a costly process and with the advent of 
’ Do not Call’ registries in countries such as India, it now faces some regulatory chal- 
lenges as well. Moreover, a significant number of customers now prefer to use the 
websites of the organisations to seek information and carry out transactions. Thus, 
it is necessary to have automatic recommendation systems that would aid in cross 
selling in an online environment. Majumdar and Mukherjee [14] describe the use 
of online cluster analysis in determining the most appropriate offers to be given to 
a customer based on the available information. They convert the characteristics of 
each customer into a vector. For each customer visiting the website, they compute 
the cosine distance of this customer with all the customers in their training dataset 
based on the information about the customer available with them. The cosine dis- 
tance between two customers having vector representation x and y is given by an 7 
where < xX, Ÿ > is the dot product between the two vectors x and ÿ and the |.| denotes 
the length of a vector. Note that the cosine distance takes the maximum value of 
+1 when x = yj and hence is not a distance in the usual sense. In what follows I use 
the term cosine similarity instead of cosine distance. The products purchased by the 
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‘micro-cluster’ of customers in the training dataset having the maximum (or close to 
the maximum) cosine similarity with the current customer are analysed. Each prod- 
uct is broken down in terms of their attributes and each product attribute is given a 
weight. Suppose there are m product attributes and n customers in the micro-cluster. 
A simple weighting scheme is to give the attribute A; the weight w,, where 


i din Sd@ 
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Here f; is the frequency of occurrence of attribute A; among the purchases of the 
ith customer in the micro-cluster and d(i) is the cosine similarity of the ith customer 
with the current customer. The current customer is then recommended products that 
have the attributes that have been assigned the highest weights. One of the problems 
that would be faced by this method would be that it would require frequent updating 
of the training dataset as the customers’ buying preferences is likely to change over 


time. 


5.2 Talent Acquisition 


Large service organisations in the IT-sector in India need to recruit very large num- 
ber of people every year to maintain their growth and account for attrition. Tata Con- 
sultancy Services (TCS) which is a large company having more than 300,000 lakh 
employees recruited approximately 70000 new staff in the fiscal year 2012-2013. 
This requires huge investment in attracting the right kind of people to apply, screen- 
ing these applications, interviewing them, selecting the most deserving of them, cre- 
ate offers which attract the selected candidates to join the organisation and finally 
train them and make them ready for their job. Srivastava [25] discusses the use of 
analytics for improving the talent acquisition process at TCS with the goal of getting 
the required number of good quality new hires at a reduced overall cost meeting the 
lead time specification for each role. Working on a dataset of 26574 candidates who 
were interviewed by TCS-BPO during the year 2010 they use decision trees to predict 
(1) whether a candidate will be selected if interviewed and (11) if a selected candidate 
will join the organisation. They use linear regression modelling to predict the time 
of joining of a selected candidate and also carry out “subgroup discovery’ to identify 
the characteristics of candidates who take much longer time to join compared to the 
others. Further, since offer declines are very costly a Bayesian net is used to find out 
the root cause of the offer declines, so that by addressing these causes offer declines 
could be reduced. 

It is not difficult to see that the talent acquisition process described above can be 
further improved if unstructured data such as audio/video recordings of interviews, 
statements about projects worked on, past experience, career objectives, etc., made in 
the candidate’s resume (or application form) could be used for model development. 
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However, better methods of extracting information from unstructured data coming 
from multiple sources would be required to make this possible. 


6 Concluding Remarks 


In this paper, we have discussed some of the statistical challenges that are faced by 
statisticians and data scientists while doing analytics with Big Data. These challenges 
give statisticians significant opportunities for developing new statistical methods that 
(a) are scalable and can handle large volumes of data easily, (b) analyse stream- 
ing data and give ‘Good answers fast’, (c) handle multiple data types and help in 
understanding association amongst data of different types, (d) can derive meaning- 
ful results even with poor quality data and (e) protect privacy and confidentiality of 
individual level data without impacting the accuracy of aggregate level analysis. We 
have also briefly discussed two methods Symbolic Data Analysis and Approximate 
Stream Regression, which holds promise in addressing some of the challenges with 
Big Data. In addition, two case studies of applications of analytics in management— 
one in marketing management and the other in human resource management—are 
discussed. 
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Application of Mixture Models to Large 
Datasets 


Sharon X. Lee, Geoffrey McLachlan and Saumyadipta Pyne 


Abstract Mixture distributions are commonly being applied for modelling and for 
discriminant and cluster analyses in a wide variety of situations. We first consider 
normal and f-mixture models. As they are highly parameterized, we review meth- 
ods to enable them to be fitted to large datasets involving many observations and 
variables. Attention is then given to extensions of these mixture models to mixtures 
with skew normal and skew tf-distributions for the segmentation of data into clusters 
of non-elliptical shape. The focus is then on the latter models in conjunction with 
the JCM (joint clustering and matching) procedure for an automated approach to the 
clustering of cells in a sample in flow cytometry where a large number of cells and 
their associated markers have been measured. For a class of multiple samples, we 
consider the use of JCM for matching the sample-specific clusters across the sam- 
ples in the class and for improving the clustering of each individual sample. The 
supervised classification of a sample is also considered in the case where there are 
different classes of samples corresponding, for example, to different outcomes or 
treatment strategies for patients undergoing medical screening or treatment. 


Keywords Flow cytometry * Sample of cells + Multiple samples - Clustering of 
cells + Supervised classification of samples > Skew mixture models - EM algorithm 


1 Introduction 


Finite mixture distributions are being increasingly used to model observations on 
random phenomena in a wide variety of applications. Many of these applications 
use normal mixture models, which can be fitted easily at least for independent data 
using the expectation—maximization (EM) algorithm of Dempster et al. [1]. This 
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seminal paper greatly simulated interest in the use of finite mixture distributions to 
model heterogeneous data. This is because the fitting of mixture models by maximum 
likelihood (ML) is a classic example of a problem that is simplified considerably by 
the EM’s conceptual unification of ML estimation from data that can be viewed as 
being incomplete; see, for example, McLachlan and Peel [2]. 

One field in which the normal mixture model and its robust version using 
t-distributions as components have been applied is bioinformatics with particular 
attention to the analysis of microarray gene-expression data. This has involved prob- 
lems such as the unsupervised classification (cluster analysis) of so-called tissue 
samples containing the expression levels of thousands of genes [3]. As the normal 
and t-mixture models are highly parametrized, consideration has to be given to ways 
in which they can still be fitted to big datasets in which the number of variables p is 
usually very large relative to the sample size n. 

We shall give a brief account of the fitting of normal and f-mixtures to high- 
dimensional data, including the use of mixtures of factor models in which the 
component-covariance matrices are taken to have a factor-analytic representation. In 
some applications, the number of variables are too large for even mixtures of factor 
models to be fitted. Hence some form of dimension reduction has to be undertaken. 
We shall consider methods such as variable selection and projections of the data into 
a space of much lower dimension than p. 

More recently, mixture models are being used in an attempt to provide an auto- 
mated approach to the analysis of samples in flow cytometry. Flow cytometry is a 
powerful tool in clinical diagnosis of health disorders, in particular immunological 
disease. It offers rapid high-throughput measurements of multiple characteristics on 
every cell in a sample, capturing the fluorescence intensity of multiple fluorophore- 
conjugated reagents simultaneously. It is routinely used in clinical practices for the 
diagnosis of health disorders such as immunodeficiency diseases. 

A critical component in the pipeline of flow cytometry data analysis is the iden- 
tification of cell populations from a multidimensional dataset that consists of a very 
large number of cells (perhaps in the hundreds of thousands). Hence, there is the 
need to segment the cells into a number of clusters on the basis of markers mea- 
sured on each cell. In the case of multiple samples of cells, there is the additional 
registration problem of matching the sample-specific clusters across the samples. In 
situations where there are different classes of samples, a further problem concerns 
the construction of a classifier for assigning an unclassified sample to one of the pre- 
defined classes. The focus is on the JCM (joint clustering and matching) procedure 
for carrying out these aforementioned tasks. This procedure was proposed by Pyne 
et al. [4] to enable the clustering and registration problems to be performed simul- 
taneously rather than first clustering each sample and then having the problem of 
trying to match the sample-specific clusters as in the FLAME procedure proposed 
earlier by Pyne et al. [5]. With the JCM approach to registration for multiple sam- 
ples and supervised classification in the case of a number of predefined classes of 
samples, a template is constructed for each class. 

To establish some notation, we let Y denote a p-dimensional random vector con- 
sisting of p feature variables associated with the random phenomenon of interest. 
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We let also y,, ..., y, denote the n observed samples on Y. We shall refer to each y, 
as a feature vector. 
The n X p data matrix A can be expressed as 


A= Gess) (1) 


For example, in the context of analyzing gene-expression data from n microarray 
experiments on p genes, the feature vector y, contains the expression levels of the p 
genes resulting from the jth microarray experiment (j = 1, ..., n). Usually, p is very 
large (in the thousands or tens of thousands) compared to n, which is often less than 
a hundred. The feature vector y, is known in this context as the gene signature vector 
while the ith row of the data matrix A is referred to as the profile (vector) for the ith 
gene. 

In the analysis of flow cytometry data, the feature vector y; contains the measure- 
ments on p markers on the jth cell. In this application n is usually very large, for 
example, in the hundreds of thousands, while in the past the number of markers p 
would be small (say, less than five), which enabled manual gating to be undertaken. 
Currently available flow cytometers can measure up to about 20 markers simultane- 
ously, and so there is a need for an automated approach. The next generation of mass 
cytometers are able to measure more than 40 markers. 


2 Finite Mixture Models 


With the mixture model-based approach to density estimation and clustering, the 
density of Y is modelled as a linear combination (or mixture) of g component den- 
sities f(y) with unknown mixing proportions m; (i= 1..., T). That is, each data 
point y; is taken to be a realization of the probability density function (p.d.f.) of the 
mixture model, given by 


g 
fo) = aio)  G=1,. n). (2) 
i=] 


The mixing proportions must satisfy the conditions z, > 0 and 2 m; = 1, that is, 
they are nonnegative and sum to one. 

In density estimation, the number of components g in the mixture density (2) can 
be taken to be sufficiently large to provide an arbitrarily accurate approximation of 
the underlying density function; see, for example, Qi and Barron [6]. For clustering 
purposes, each component in the mixture model (2) is usually taken to correspond to 
the distribution of the data in a single cluster. Then a probabilistic clustering of the 
data into g clusters can be obtained in terms of the fitted posterior probabilities of 
component membership for the data. The posterior probability that an observation 
with feature vector y, belongs to the ith component of the mixture is given by 
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TiO) = mfi Y). (3) 


On specifying a parametric form f;(y;; 0;) for each component density, the para- 
metric mixture model 


g 
fo D= È afi; 4) (4) 
i=1 


can be fitted by maximum likelihood (ML). In the above, 0; is the vector consisting 
the unknown parameters of the ith component of the mixture model. The vector 
Y = (éf, Dip tes 4) contains the unknown parameters of the mixture model, 
where & consists of the elements of the 0; known a priori to be distinct; the superscript 
T denotes vector transpose. We let Q denote the parameter space for Y. 

The maximum likelihood estimate (MLE) of ¥, denoted by W, is given by an 
appropriate root of the likelihood equation. That is, W is a solution for W to the 
equation 

d log L(W)/oW = 0, (5) 


where L() denotes the likelihood function for W, given by 


LW) = | [f0 ©). 
j=] 


J 


which is formed under the assumption of independence of the data. Solutions of the 
likelihood equation (5) corresponding to local maximizers of log likelihood func- 
tion log L(W) can be obtained via the expectation-maximization (EM) algorithm of 
Dempster et al. [1]; see also McLachlan and Krishnan [7]. 

In practice, the clusters in the case of Euclidean data are frequently assumed to be 
elliptical, so that it is reasonable to consider fitting mixtures of elliptically symmetric 
component densities. Within this class of component densities, the multivariate nor- 
mal density is a convenient choice given its computational tractability. In this case, 
the ith component density f,(y;; 0;) is given by 


fiO) = pOH) =l, ..., g), (6) 


where #(y,; Hi, &;) denotes the p-dimensional multivariate normal distribution with 
mean vector u; and covariance matrix &,. The normal mixture density is then spec- 
ified as 


g 
FO) = È rp; He E). (7) 
i=1 


The normal mixture model (7) is sensitive to outliers in the data since it adopts 
the multivariate normal family to model the distributions of the errors. To improve 
the robustness of this model for the analysis of data, which have longer tails than 
the normal or contain atypical observations, an obvious way is to consider using a 
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heavy-tailed distribution such as the multivariate t-distribution. The f-distribution, 
along with some other well-known distributions such as the Cauchy distribution and 
Pearson type VII distribution, can be obtained by embedding the normal distribution 
in a wider class of elliptically symmetric distributions. With the ¢-distribution, there 
is an additional parameter v; known as the degrees of freedom [8]. This parameter 
v; may be viewed as a robustness tuning parameter that regulates the thickness of 
the tails of the distribution. When fitting the t-mixture model, this parameter can be 
fixed in advance (that is, known a priori) or it can be inferred from the data as part 
of the model fitting process. 

One attractive feature of using mixture models that adopt elliptically symmet- 
ric distributions as components densities (such as the aforementioned normal and 
t-mixture models) is that the implied clustering is invariant under affine transforma- 
tions. Also, in the case where the components of the mixture correspond to externally 
defined subpopulations, the unknown parameter vector W can be estimated consis- 
tently by a sequence of roots of the likelihood equation. Concerning the issue of 
how to select an appropriate number of components for a mixture model, this can be 
approached by consideration of the value of the log likelihood function. For exam- 
ple, one may consider performing model selection using criteria such as the Bayesian 
Information Criterion (BIC) of Schwarz [9]. Another method is to employ a resam- 
pling approach such as [10] to test for the smallest value of g compatible with the 
data. 


3 Factor Models 


For datasets where the dimension p of the observation or feature vector is large rela- 
tive to the number of observations n, the normal mixture model (7) may not be able 
to be fitted directly to the observed data. This is due to the normal mixture model (7) 
with general component-covariance matrices being a highly parameterized model. 
With unrestricted covariance matrices, there are > p(p + 1) distinct elements in each 
2. A simple way of proceeding in such situations would be to take the 2; to be diag- 
onal. However, this leads to the restriction that the axes associated with the clusters 
must be aligned with those of the feature space, which may not be appropriate in 
practice as the clusters are typically of arbitrary orientation. Further restricting the 
x, to be a multiple of the identity matrix leads to spherical clusters. If the mixing 
proportions are taken to be the same, then it leads to a soft-version of k-means. 

A common approach to reducing the dimensionality of the data is to perform 
a principal component analysis (PCA). The latter, however, provides only a global 
linear model for the representation of the data in a low-dimensional subspace. A 
more general and powerful approach is to adopt a mixture of factor anlayzers (MFA), 
which is a global nonlinear approach obtained by postulating a finite mixture of linear 
(factor) submodels for the distribution of the full observation vector y j given a rela- 
tively small number of (unobservable) factors (McLachlan and Peel [11], Chap. 8). 
That is, a local dimensionality reduction method is provided by imposing the factor- 
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analytic representation 
x, = BB) +D, (8) 


In the above, B; is a p X q matrix known as the factor loadings and D, is a diagonal 
matrix (i= 1, ..., g). 

In practice, there is often the need to reduce further the number of parameters 
in the factor analysis model described above. Further restrictions can be applied to 
the specification of the elements of the component-covariance matrices. To this end, 
Baek and McLachlan [12] proposed the use of the so-called mixtures of common 
factor analyzers which uses common component-factor loadings that specifies the 
distribution of the feature vector Y, conditional on its membership of component i 
as 

Y, = BU, + e;, (9) 


where B is a p x q matrix of common factor loadings and U,,, ..., U,,, are distrib- 
uted independently N(0,/,). It was so-named mixtures of common factor analyzers 
(MCFA) since the matrix B of factor loadings in (9) is common to all g components 
before the transformation of the factors U;; to be white noise. This approach con- 
siderably reduces further the number of parameters in the MFA model. Moreover, it 
allows the data to be easily visualized in low dimensions by plotting the correspond- 
ing (estimated) factor scores. 

Similar to the normal mixture model, the mixture of (normal) factor analyzers 
model is not robust to outliers since it uses the normal distribution for the errors and 
factors. In Baek and McLachlan [13], the so-called mixture of t-analyzers (MCtFA) 
was introduced in an attempt to make the model less sensitive to outliers, which 
replaces the normal distribution in the MCFA model with its more robust version— 
the multivariate ¢-distribution. 


4 Dimension Reduction 


When it is computationally feasible to fit the factor models as defined in the previous 
section, dimension reduction is effectively being done as part of the primary analysis. 
However, for many datasets in data mining applications, the number of variables p 
will be too large to fit these models directly without first performing some form of 
dimension reduction. 


4.1 EMMIX-GENE 


In the context of clustering, McLachlan et al. [14] proposed a three-step procedure 
(called EMMIX-GENB), in which in the first step the variables are considered indi- 
vidually by performing a test of a single t-component distribution versus a mixture of 
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two t-components for each variable. Variables found not to be significant according 
to this test are discarded. Then on the second step, the retained variables (after appro- 
priate normalization) are clustered into groups on the basis of Euclidean distance. 
Finally, on the third step, the observations can be clustered by the fitting of mixtures 
of normal or f-distributions or factor analyzers to representatives of the groups of 
variables. 

Chan and Hall [15] have considered a method for variable selection where the 
variables are considered individually by performing a nonparametric mode test to 
assess the extent of multimodality. 


4.2 Projection Methods 


Another approach to dimension reduction is projection, which is very popular. For 
example, principal component analysis (PCA) is commonly used as a first step in the 
analysis of huge datasets, prior to a subsequent and more indepth rigorous modelling 
of the data. A related projection method is matrix factorization, which is described 
briefly in the next section. 


4.3 Matrix Factorization 


In recent times, much attention has been given to the matrix factorizations approach 
for dimension reduction. With this approach, the data matrix A as defined by (1) can 
be expressed as 

A=A,A), (10) 


where the matrices A, and A, is an x q matrix and a q X n matrix and, respectively. 
Here g is chosen to be much smaller than p. Given a specified value of g, the matrices 
A, and A, are chosen to minimize the function 


IA —A,A,||’. (11) 


Dimension reduction is effected by replacing the data matrix A by the solution A, for 
the factor matrix A,, where the jth row of A, gives the values of the g metavariables 
for the jth sample. This implies the original p variables are replaced by g metavari- 
ables. In the case where the elements of A are nonnegative, we can restrict the ele- 
ments of A, andA, to be nonnegative. This approach is known as nonnegative matrix 
factorization (NMP) in the literature [16, 17]. We shall call the general approach 
where there are no constraints on A, and A,, general matrix factorization (GMF). 
The classic method for factoring the data matrix A is singular-value decompo- 
sition (SVD); see Golub and van Loan [18]. It follows from this theorem that the 
value of A, that minimizes (11) over the set of all n X q matrices of rank q is given 
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by the matrix whose rows are the the eigenvectors corresponding to the g largest 
eigenvalues of A’ A. 

Note that with SVD, effectively PCA, it imposes orthogonality constraints on the 
rows of the matrix A,. Thus this ignores the possible nonindependence of the vari- 
ables measured, for example, the genes in the biological processes under study; see, 
Kossenkov and Ochs [19]. On the other hand, the GMF approach has no constraints 
on C,. Thus it provides a factorization of A into a low-dimensional subspace without 
any orthogonality constraints on its basis vectors. Hence, GMF is more flexible for 
modelling, for example, biological behaviour in which the gene signatures overlap. 
In contrast, due to its orthogonality constraints, PCA is overly restrictive for such 
data and is thus not suitable for the task of isolating gene signatures that have appre- 
ciable overlap. Furthermore, PCA is based on finding the directions of maximum 
variance, but the sample covariance matrix may provide misleading estimates where 
the number of variables p is much greater than the number n of observations [20]. 

Nikulin and McLachlan [21] have developed a very fast approach to the gen- 
eral matrix factorization (2), using a gradient-based algorithm that is applicable 
to an arbitrary (differentiable) loss function. Witten et al. [22] and Nikulin and 
McLachlan [23] have considered a penalized approach to PCA in order to provide 
Sparse solutions. 


5 Flow Cytometry Data 


Data from a flow cytometry sample comprise parallel measurements of multiple 
physical, chemical and biological characteristics and properties of each individual 
cell. Briefly, cells in a sample are stained with a number of fluorophore-conjugated 
antibodies (or markers) before they are sequentially passed through laser beams in a 
flow cytometer that excite the fluorochromes. The light that emerges from each cell is 
captured and quantitated by a series of fluorescence detectors. These measurements 
are collected at a high speed, typically several thousands of cells per second, thus 
generating rich and complex high-resolution data in a high-throughput manner. In 
flow cytometric analysis, these simultaneous measurements of multiple fluorescent 
intensities can be used to study the differential expression of different surface and 
intracellular proteins, enabling biologists to study a variety of biological processes 
at the cellular level. 

Traditionally, data are analyzed manually by trained analysts. The data are sequen- 
tially projected onto a series of bivariate plots and regions (gates) of interests are 
identified by the analyst manually, a process known as gating. With the advancement 
in technology allowing over 20 markers to be measured simultaneously, this has sur- 
passed the practical feasibility of conventional manual analysis. Due to this and the 
rather the subjective nature of this approach, much recent efforts have focused on the 
development of computational tools to automate the gating process and to perform 
further analysis of big flow cytometry data including cluster alignment and super- 
vised classification; see Aghaeepour et al. [24] for a recent account. 
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Mixture models provide a convenient and formal framework for modelling these 
complex and multimodal data. Under the mixture model framework, the flow cytom- 
etry data can be conceptualized as a mixture of populations, each of which consists of 
cells with similar expressions. The distribution of these cell populations can be char- 
acterized mathematically by a parametric density. Thus, the task of cell population 
identification translates directly to the classical problem of multivariate model-based 
clustering, where each cell population corresponds to a cluster of points with similar 
fluorescence intensities in the multidimensional space of markers. 


5.1 Non-elliptical Clusters 


It is well known that data generated from flow cytometric studies are often asymmet- 
rically distributed, multimodal, as well as having longer and/or heavier tails than nor- 
mal. To accommodate this, several methods adopted a mixture of mixtures approach 
where a final cluster may consist of more than one mixture component; see, for exam- 
ple, the procedures SWIFT [25], HDPGMM [26] and ASPIRE [27]. Another proce- 
dure, flowClust [28, 29], applied data transformation techniques to handle asymmet- 
ric clusters prior to using traditional mixture models. Further efforts have focused on 
the application of mixture models with skew distributions as components to enable 
a single component distribution to correspond to a cluster [4, 5, 30]. To this end, 
both FLAME [5] and JCM [4] adopt component densities that are a skew version of 
the ¢-distribution, which can be expressed as 


F00 = 21,9; uE, Vv) r, (4:0,1,v+p), (12) 


where q = TE! (y — p) TE A = 1 — TE! 8, dy) = (y — WTE! (y — p) and 
6 is p-dimensional vector of skewness parameters. In (12), ty; H, &, v) denotes the 
p-variate t-density with location y, scale matrix & and degrees of freedom v, and T, 
denotes its corresponding distribution function. The f-distribution allows for heavier 
tails than the normal distribution, thus providing a more robust approach to tradi- 
tional normal mixture modelling. The skew version (12) of the f-distribution, with 
an additional vector of skewness parameters 6, allows for non-symmetric distribu- 
tional shapes. This version after an appropriate transformation is equivalent to the 
model proposed by Azzalini and Capitanio [31]. In recent years, many different ver- 
sions of the skew f-distribution have been proposed; see [32] for an overview on this 
topic. Parameter estimation for various versions of skew normal and skew f-mixture 
models can be carried out using the EM algorithm [33, 34]. 
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5.2 FLAME Procedure 


The FLAME procedure was developed as a platform for automated gating (or cluster- 
ing) and matching of flow cytometric data across samples and classes. With FLAME, 
each sample of cells can be effectively modelled by a mixture of skewed and heavy- 
tailed multivariate distributions. Segmentation of cells then corresponds to cluster- 
ing based on the fitted skew mixture model. In the case of multiple samples (from 
the same cohort), the problem of matching cell populations across samples is han- 
dled by FLAME using a two-stage approach by undertaking Partitioning Around 
Medoids (PAM) and bipartite matching (BP). Firstly, a template consisting of the 
location of meta-clusters is constructed using PAM, based on the modes of all the 
populations identified by the mixture model in each sample. Subsequently, the pop- 
ulation modes in each sample are matched to the template meta-clusters using a BP 
algorithm, which takes into account the proportions of cells in each population. This 
latter step allows extra or missing populations to be detected, and spurious popula- 
tions to be recombined. The post-hoc steps are applicable not only to within-class 
matching of populations, but can also be directly applied to match meta-clusters 
across different classes. 


5.3 JCM Procedure 


In addition to the problem of clustering and matching of cell populations, the unsu- 
pervised classification (clustering) of unlabelled samples presents another challeng- 
ing task in the pipeline of flow cytometric data analysis. While FLAME was pri- 
marily designed for the first two task, no overall parametric template was formed to 
describe the characteristics of the entire batch or class of samples and no algorithm 
was provided for the classification of unlabelled samples. Further, the matching of 
populations across samples is performed in a post-hoc fashion independent of the 
clustering step, and the accuracy of the PAM is sensitive to the amount of inter- 
sample variation in a batch. 

Pyne et al. [4] proposed the multi-level JCM framework to address these concerns 
by operating on an entire batch of samples together. This enables the clustering and 
matching of populations to be performed simultaneously. Similar to FLAME, the 
JCM procedure model each populations with a component distribution. However, 
JCM introduced an additional higher level to link the individual mixture models from 
each sample to an overall parametric template. It is achieved by adopting a random- 
effects model (REM) to handle the inter-sample variations, where every sample in a 
given batch can be viewed as an instance of the batch template possibly transformed 
with a flexible amount of variation. An advantage of this approach is that each cluster 
in a sample is automatically registered with respect to the corresponding cluster of 
the template. A further advantage is the availability of an overall parametric template 
for the batch, which facilitates quantitative comparisons across different batches of 
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samples. It also provides a basis for the allocation of an unlabelled sample to a class, 
where the batches represent training data from each different class. This latter task is 
handled by JCM by comparing the Kullback—Leibler (KL) divergence from the class 
template [35, 36]. Specifically, once a mixture model is fitted to an unlabelled sam- 
ple, its label is predicted to be the class associated with the smallest KL divergence. 


5.4 Clustering and Matching of Flow Cytometric Samples 


To demonstrate the FLAME and JCM procedures in clustering and aligning cell 
populations in a batch of samples, we consider the diffuse large B-cell lymphoma 
(DLBCL) benchmark dataset from the flowCAP-I challenge (Aghaeepour et al., 
2013). For this illustration, we use the subset of 16 samples which were manually 
analyzed and were determined to have the same number of clusters. These samples 
were derived from the lymph nodes of patients diagnosed with DLBCL, each stained 
with three markers—CD3, CD5 and CD19. Together with the forward scatter and 
side scatter measurements, each sample is represented by an X p expression matrix 
with p = 5 and the number cells, n, ranging from 3000 to 25000. 

Focusing first on the gating of these samples, we applied FLAME and JCM to 
cluster the 16 DLBCL samples. The FLAME procedure models each sample indi- 
vidually by fitting a mixture of skew f-distributions. With JCM, a template is created 
across the batch of 16 samples and each sample is linked to the template through an 
affine transformation. A skew t-mixture model is then fitted to each sample via the 
EM algorithm, using the JCM template as the initial values. The effectiveness of JCM 
and FLAME can be evaluated by comparing their clustering results with the results 
given by manual gating. The misclassification rate (MCR) and F-measure values for 
each sample are reported in Table | of Pyne et al. [4]. Note that a lower MCR and 
a higher F-measure indicate a closer match between the labels given the algorithm 
and those given by manual gating. The reported average MCR for JCM and FLAME 
across the entire batch of samples was 0.0711 and 0.1451, respectively. The cor- 
responding average F-measures were 0.9403 and 0.9021. They compare favourably 
with the results for competing algorithms such as HDPGMM, flowClust and SWIFT, 
which obtained an average MCR of 0.2038, 0.1393 and 0.1454, respectively. These 
three algorithms were applied to this dataset as follows. The HDPGMM procedure 
fits a hierarchical mixture of Gaussian distributions to the dataset, where all sam- 
ples share the same model parameters except for the mixing proportions. Hence, this 
model assumes that the location and shape of the cell populations are identical across 
all samples, and the weight or proportion of these populations is the only source of 
inter-sample variations. For flowClust and SWIFT, their models were applied to each 
sample individually. 

In the case of alignment of cell populations across samples, FLAME can match 
the fitted component densities with the post-hoc steps described in Sect. 5.2. Starting 
from the models fitted to each samples (Fig. la) as produced by the clustering step 
above, FLAME runs PAM on the modes of these component densities to obtain 
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Table 1 Average MCR of various algorithms on the DLBCL dataset 


Algorithm FLAME HDPGMM SWIFT 
AMCR 0.0711 0.1451 0.2038 0.1151 0.3946 


meta-clusters (Fig. 1b). The populations in each sample are then matched to these 
meta-clusters by undertaking BP (Fig. 1c). With JCM, the clustering and alignment 
of populations are performed simultaneously through the fitting of a JCM template 
(Fig. 1d). 
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Fig. 1 Clustering and Matching of 16 DLBCL samples by FLAME and JCM. a FLAME cluster- 
ing: a skew f-mixture model is fitted to each sample individually. b FLAME meta-clustering: the 
modes of all clusters from each sample were pooled and clustered using PAM to construct a tem- 
plate of meta-clusters. c FLAME populations matching: clusters from each sample were matched 
to meta-clusters using BP. d JCM: clustering and matching are performed simultaneously by fit- 
ting a random-effects model where each sample is linked to a template model through an affine 
transformation 
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5.5 Classification of Flow Cytometric Samples 


In a recent study on germinal centre (GC) lymphoma, Craig et al. [37] studied cellu- 
lar hierarchies that may help distinguish germinal centre hyperplasia (GC-H) from 
germinal centre B-cell lymphoma (GC-L). Lymphoid tissue biopsy specimens from 
100 individuals were analyzed using flow cytometry with markers CD45, CD20, 
kappa, lambda, CD19, CD5, CD10 and CD38. Several other measurements were also 
recorded for each cell, including forward and side scatter measurements, resulting in 
a collection of 100 data files, each with 15 variables and around 30,000 observations. 
These samples were also manually analyzed to obtain training data. There were 48 
cases of GC-L and 52 cases of GC-H. For our illustration, we allow JCM to train on 
70 samples (consisting 37 cases of GC-L and 33 cases of GC-H), and evaluate the 
classification performance of JCM on the remaining 30 samples. 

We applied the JCM procedure and six other algorithms (ASPIRE, Citrus, flow- 
Match, flowPeaks, SWIFT and HDPGMM,) to this dataset. Since the latter three 
algorithms do not provide a strategy for sample classification, we follow a simi- 
lar approach to Dundar et al. [27] where a support vector machine (SVM) classi- 
fier is built based on the mixing proportions of the template model applied to each 
sample. For SWIFT and flowPeaks, the template model is computed by pooling all 
samples together. For HDPGMM, a consensus model is obtained by modal clus- 
tering. With JCM, ASPIRE, flowPeaks, flowMatch, SWIFT and HDPGMM, their 
templates identified 9, 2, 11, 4, 8 and 16 clusters, respectively. Table 2 reports some 
common measures of classification performances for these algorithms on the GC 
dataset, including sensitivity, specificity, accuracy, precision and F-measure. Among 
these algorithms, JCM achieved the highest F-measure value for this dataset, as well 
the highest accuracy. Both JCM and HDPGMM yield perfect specificity and preci- 
sion. On comparing the scores or measures used by the classifiers of these various 
methods (Fig. 2), the classification score adopted by JCM has the best classification 
performance for this dataset. The classification scores for each of the methods is 
calculated as follows. In Fig. 2a, the KL score of JCM is calculated as the KL dis- 
tance from the GC-L class over the sum of the KL distance from the GC-L and GC-H 
classes. Thus, a higher score means that the model fitted to the sample is closer to the 
GC-L template. For SWIFT, flowPeaks and HDPGMM (Fig. 2b, d, f respectively), 
the decision values given by the SVM classifier are shown. In Fig. 2c, a classification 
score for the Citrus procedure can be obtained by calculating the proportion of Citrus 
classifiers that predicted GC-L for a sample. Figure 2e shows the scores computed 
by flowMatch; see Azad et al. [38] for details. 


6 Summary and Conclusions 


We have considered how mixture models can be applied to large datasets involving 
many observations and/or variables. Two real examples in flow cytometry are pre- 
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Table 2 Classification results of various algorithms on the GC dataset 


iw 088 


sented. For high-dimensional datasets where traditional normal and t-mixture mod- 
els cannot be fitted directly, mixtures of factor models provide a computationally 
less demanding alternative, whereby the full observation vector is modelled by pro- 
jecting onto low-dimensional spaces. Further reduction in the number of parameters 
to be estimated can be achieved using factor models with common factor loadings. 
In cases where even factor models may not be computationally feasible, dimension 
reduction techniques such as variable selection, projections and matrix factorization, 
can be applied prior to the fitting of mixtures of factor analyzers. 

Recent advances in flow cytometry technology have enabled rapid high- 
throughput measurements to be taken, generating big and high-dimensional datasets 
for analysis. This has sparked the development of various computational tools for 
automated analysis of flow cytometry samples. Among these methods, mixture mod- 
els have been widely adopted as the underlying mechanism for characterizing the 
heterogeneous cell populations in these cytometry samples; see, for example the pro- 
cedures FLAME [5], SWIFT [25], flowClust [28], flowPeaks [39], HDPGMM [26], 
ASPIRE [27], JCM [4] and the articles [30, 40—42]. Observing that these clusters 
of cells in flow cytometry samples are typically not elliptically symmetric and may 
contain outliers, one approach is to consider the fitting normal mixture models with 
a large number of components and then subsequently merging some of the compo- 
nents. However, this entails the problem of trying to identify the correspondence 
between components and clusters. To tackle this, FLAME adopted mixture models 
with skew normal and skew t-component densities that can flexibly accommodate 
asymmetrically distributed clusters with tails that may be longer than normal. In the 
analysis of a cohort of flow cytometry data, consisting of multiple samples from dif- 
ferent classes that may corresponds to different clinical outcomes, one is also inter- 
ested in the classification of unlabelled samples to a predefined class. To this end, 
some of the above mentioned procedures have been extended to perform alignment 
of cell populations across multiple sample and the construction of a template to rep- 
resent the characteristics of a batch of samples; for example, ASPIRE and JCM. This 
facilitates (supervised) classification of new samples. These two methods introduced 
random-effects to cater for inter-sample variations, with the class template of the for- 
mer procedure based on hierarchical mixtures of normal distributions, while the lat- 
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Fig. 2 Plots displaying the scores or measures used by different classifiers for the GC dataset. Blue 
triangles denote a GC-H sample, whereas red dots denote a GC-L sample. A dash line indicate the 
classification boundary used by a classifier. a (JCM) scatter plot of the KL score by JCM, given by 
the relative KL distance from the GC-L class. Samples corresponding to points that lie above the 
boundary (dashed line) are likely to be GC-L. b (SWIFT) scatter plot of the scores given by the 
SVM classifier. The threshold is optimized based on the training data. Note that samples are ordered 
in ascending order of SVM scores. c (ASPIRE) scatter plot of the SVM classifier. d (Citrus) scatter 
plot of the classifier score for each sample (based on the results of the 100 regression classifiers 
selected by Citrus). Samples corresponding points that lie above the boundary are classified as GC- 
H. e (flowPeaks) scatter plot of the scores given by the SVM classifier. f (flowMatch) scatter plot 
of scores 
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ter is based on mixtures of skew t-distributions. The effectiveness of JCM was illus- 
trated on a germinal centre lymphoma dataset, in which JCM compares favourably 
with other available model-based methods and non-model-based methods. 
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An Efficient Partition-Repetition Approach 
in Clustering of Big Data 


Bikram Karmakar and Indranil Mukhopadhayay 


Abstract Addressing the problem of clustering, 1.e. splitting the data into homoge- 
neous groups in an unsupervised way, is one of the major challenges in big data ana- 
lytics. Volume, variety and velocity associated with such big data make this problem 
even more complex. Standard clustering techniques might fail due to this inherent 
complexity of the data cloud. Some adaptations are required or demand for novel 
methods are to be fulfilled towards achieving a reasonable solution to this problem 
without compromising the performance, at least beyond a certain limit. In this arti- 
cle we discuss the salient features, major challenges and prospective solution paths 
to this problem of clustering big data. Discussion on current state of the art reveals 
the existing problems and some solutions to this issue. The current paradigm and 
research work specific to the complexities in this area is outlined with special empha- 
sis on the characteristic of big data in this context. We develop an adaptation of a 
standard method that is more suitable to big data clustering when the data cloud is 
relatively regular with respect to inherent features. We also discuss a novel method 
for some special types of data where it is a more plausible and realistic phenom- 
enon to leave some data points as noise or scattered in the domain of whole data 
cloud while a major portion form different clusters. Our demonstration through sim- 
ulations reveals the strength and feasibility of applying the proposed algorithm for 
practical purpose with a very low computation time. 


1 Introduction 


The fun with the concept of big data is that while its existence is undoubted, there 
cannot be any formal way to define it in the language of mathematics. It has been 
argued that the major characteristics of big data are volume, variety, velocity and 
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complexity [10]. These make big data analysis challenging and also distinguish it 
from classical data analysis. Thinking back, big data is not the only member of the 
class of concepts in data science that is not defined in mathematically precise manner. 
Clustering is another such technique which does not have any axiomatic definition. 
The task of clustering a data would be familiar to everyone but making the problem 
precise seems impossible [13]. 

Broadly, clustering has been viewed as a tool to achieve one of following three 
goals: (1) data exploration, (2) confirmation of hypothesis, and (3) preprocessing 
data to felicitate some objectives in mind. We discuss below these three aspects of 
clustering in the context of big data. This discussion should allow the reader to com- 
prehend the practical use of clustering ideology in big data analytics. 


Data exploration—Finding hidden patterns in the data is the first and foremost 
objective of data analysis. If we want to put this idea in perspective of big data one 
would be very ambitious about extracting more and more interesting patterns. This 
optimism is a powerful force in driving interesting researches on big data analytics. 
If one thinks about the problem a little more carefully, even putting aside the trou- 
ble of actually being able to handle the data, it would become evident that just an 
abundance of data is not necessarily the best thing. What is more important is the 
quality of the data and our ability of assess that quality through stable methods to 
extract quantitative information out of such data. The idea of big data itself compro- 
mises with the quality of the data. Big data accumulated from heterogeneous sources, 
Sometimes in an uncontrolled manner, introduces enormous amount of noise. This 
noise may be due to measurement error, presence of outliers, significant amount of 
missing observations, etc. Even in more hopeful scenarios like genetic data or fMRI 
data where we have a control on data generation process, noise in the data is intro- 
duced as a curse of dimensionality or due to other reasons. In such a case we would 
be looking for some small set of genes or pixels, respectively, while all other genes 
or pixels are essentially noise. 


Conformation of hypothesis—This is the exactly as it says: validating a pre- 
composed hypothesis based on the data. Specific to clustering context, if we expect 
that there is a particular clustering in the data, then we would want to check whether 
such an expectation is consistent with observed data. This is actually a pretty gener- 
ous expectation from big data analysis. We should be able to provide a satisfactory 
answer to this problem. Once validated, such hypotheses could lead to follow up 
analysis of the data or provide a guidance on further experimental data collection. 
To make the problem more precise, suppose that for a dataset collected on indi- 
vidual level, it is hypothesized that a set of features determines a cluster according to 
a relation x € S. Hence, our hypothesis says, individuals with x € S have some sort 
of homogeneity among themselves. A straightforward way to validate such hypoth- 
esis is to consider a sample of paired individuals, %, each pair having x € S and 
compare them with a bunch of pairs of individuals, Y where the individuals do not 
have any restriction on feature x. In data science there are plenty of summary statis- 
tics one can think of to carry out such comparisons. Let us consider a simple one. Let 
V, and V be the total variability of feature differences for features other than x for 


An Efficient Partition-Repetition Approach ... T 


the groups %, and Y, respectively. Deicide that the data do confirm the hypothesis if 
V > Vs. The blessing of big data is that if we repeat this decision making on quite 
a few number of samples, then with a majority vote of these repeated decisions we 
shall have a very satisfactory decision rule. The challenge then remains to implement 
an efficient algorithm to carry out this decision making. 

On the first look the method discussed above may seem incoherent to usual clus- 
tering methods. This is because while in clustering based on different similarity or 
distance measures we aim to extract homogeneous groups, here the aim is slightly 
different. Given a feature that is expected to create a homogeneous group we are ask- 
ing, if such an expectation is consistent with our similarity or distance measure. This 
practical point of view is not always addressed in clustering literature, especially in 
the context of big data analytics. 


Clustering as preprocessor—Even if the data are intended to be used for some dif- 
ferent purposes other than exploration or confirmation, clustering can still be used as 
a preprocessor so that it would make later steps more feasible. Let us put this idea in 
the context of the trend of big data analytics. Computation on big data is carried out 
on clusters of machines where in each machine a ‘part’ of the data is analysed. These 
parts are determined based on either randomization or based on sources or using a 
mapping that distributes the job into multiple machines. Thus the issue of computa- 
tional ability should be kept in mind while distributing the data to different machines. 
Now suppose our objective is variable selection. Then to avoid the challenge of spu- 
rious correlation of the data, clusters of time gapped data points is a much better 
choice than random clusters. Further, such preprocessing leads us to understand the 
nature of data in different sized bins so as to apply appropriate statistical techniques 
and analytical tools. This would also reveal any significant correlation among the 
variables. 

In this article we are going to discuss in details about clustering as a tool of data 
exploration in big data analytics. The arrangement of the sections is as follows. In 
Sect. 2, we shall state a formalism of clustering big data. Section 3 is dedicated to 
elaborate on the primary classes of big data and the distinction of clustering prin- 
ciples among them. A comprehensive literature review of big data clustering algo- 
rithms is presented in Sect. 4. In Sect. 5 two meta algorithms are presented for big 
data clustering. A broad range of simulation studies are carried out in Sect. 6 for the 
proposed meta algorithms. 


2 Formulation of Big Data Clustering 


Conceptually, clustering of big data or a very large dataset although same to that 
of clustering of a moderately sized dataset in principle, they vary in many aspects 
while constructing the clusters. One of the basic problems is due to the sheer vol- 
ume or size of big data. Usually it exceeds the RAM of a computer and hence is 
extremely difficult to apply in one go. So summarizing the results from a clustering 
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algorithm requires utmost attention; otherwise huge amount of information might be 
lost and hence the downstream analysis would be meaningless or less sensitive and 
less fruitful. 

To tackle the volume of big data, partitioning is one option but the method of such 
partitioning is important as it can be done in various ways. The easiest in theory is 
to divide the dataset at random but in many situations data may come from multiple 
sources thus adding complexity to the data cloud. Applying clustering algorithm to 
a single partition is always manageable but summarizing the results requires precise 
method and plan so that memory space could be used in an optimum way at the same 
time preserving the information obtained thus far. Use of identifier or data labels to 
the actual dataset instead of storing the whole data might be a good option towards 
fulfilling this purpose. 

Another problem is that there might be a chance that one true cluster is divided 
into two clusters due to partitioning thus increasing the number of clusters. This 
might mislead the downstream analysis of the data. So there should be a check to 
see the nature of clusters that are obtained through different machines in parallel. 
After checking, a global clustering labelling should be allotted to each data point. 
Thus the summarization process needs to be revisited based on the output of the 
clustering algorithm adapted to the specificity of big data. 

Result of clustering would be informative in finding the hidden pattern that 
includes information about the correlation structure among the data points. More- 
over, this provides an overview of the space covered by points in a cluster and the 
map of spaces for different clusters. This would naturally generate some idea about 
the future analysis that can be done using the same dataset. Thus proper preprocess- 
ing during clustering and retaining the information is essential to get more insight to 
the big data set. 


3 Principles of Big Data Clustering 


Most implementation of clustering algorithms that are considered for big data ana- 
lytics are, in a sense, applicable to large data. As mentioned above the obvious differ- 
ence between big data and simply a large or high dimensional data set is that big data 
adds a level of complexity of data storage and increase in volume of data with time. 
This complexity may be due to accumulation of data from multiple sources, among 
other reasons. Consequently a clustering algorithm should operate only locally and 
update the results sequentially. 

In a big data, the volume of the data is one of its primary characteristics and 
concerns. When we are faced with the problem of variety in the data due to hetero- 
geneous sources, we are left hopeless in any sort of use of clustering algorithms. Let 
us consider the size aspect of a big data for the moment. Even if the volume is some- 
what inconceivable in practice, there are recognizable patterns in which the volume 
is accumulated. In a structured data, we can assert that volume is accumulated in 
one of the following two patterns. 
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Fixed Dimension—A large proportion of data generated in automated manner is 
essentially of fixed and very low dimension. These are mostly user data on various 
services. The service provider accumulates data on only a few variables of interest 
and the data accumulates its volume as users are served at a very high frequency. A 
specific example of such data is traffic data which collect information on the amount 
of time taken by a traveller (with a GPS device) to travel between geographical loca- 
tions and a few other relevant covariates (e.g. mode of transportation, tweets with a 
certain hashtag, etc.). 

The particular challenges in clustering such data can be characterized by (1) immo- 
bility and (11) noise accumulation. (1) Immobility of the data means that once a dataset 
is collected it is unrealistic to expect that data would be transferred completely in 
another physical memory. Immobility is garnered since the ultra high frequency of 
data generation requires sequential storages. Practically it is unreasonable to expect 
to get all the data in one storage and process it. This imposes the restriction on a 
clustering algorithm for such a data be localized. This means that the algorithm 
processes the data on its own physical location without having it copied somewhere 
else entirely. (41) Noise accumulation on the other hand is an even more serious threat 
to analyzing this kind of data. Noise in the data is undesirable especially because we 
have only a fixed dimension. If over time amount of noise gathered is more than 
informative data, then any clustering algorithm that fails to accommodate such sce- 
nario cannot hope to learn anything useful and would be no better (or even less) than 
a random partitioning. 


High Dimension—For this class of big data the volume of the data increases more in 
terms of dimension than in its size. Broadly there can be two subclasses of such data. 
In one subclass, generally there is a cost of gathering data for each individual. Popular 
examples are gene expression data and medical imaging data where data of very 
high dimension are collected only on a few individuals. In another subclass of data, 
individual data are generated over time. One such example is consumer purchase 
data collected over time by some big vendor or retailer. While in the second subclass 
the number of individuals can grow as well; such growth (in number of customers) is 
expected to be much smaller than growth in dimension (information on purchases). 

Interestingly, for this class of data we might be looking for either cluster of indi- 
viduals or cluster of variables based on the subclass we are in. In the first of the 
two subclasses, we are aiming for groups of variables that have special patterns. The 
principle challenge is to find the most relevant groups/clusters. Thus all the variables 
(e.g. genes in a gene expression data) may not be of any relevance and may be termed 
as “noise” in the dataset. Immobility of the data is less serious problem in process- 
ing this type of data. A subsampling-based algorithm is suitable for analyzing such 
data. In the second subclass of data where we are gathering individual level data over 
time, we would be interested in grouping individuals based on their pattern. In such 
a scenario, clustering is more important as a preprocessing tool. Then challenge is 
posed by spurious correlation among the variables. Dimension reduction methods 
might work as an essential tool for clustering such data. Some of basic principles of 
clustering big data can be summarized in Table 1 below. 
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Table 1 A summary: principles for clustering big data based on different data classes 


Structured data (no variety) 


Huge dimension Increasing number of 
Dimension 
Source of volume High frequency of Dimension Dimension 
observations 
Example traffic data, customer | Gene expression data, | Consumer data 
survey data medical imaging data 
Objective/goal Cluster individuals Cluster relevant Clustering individuals 
variables of similar pattern 
Challenge Immobility of data and | Too many irrelevant | Immobility of data and 
noise accumulation variables spurious correlation 
Feature Growing number of Irrelevant variables 
clusters work as noise 
Principle Localized algorithms, | Subsampling-based Localized algorithm, 
accommodate for algorithm dimension reduction 


noise, sequential 
updating 





Thus volume aspect provides us with quite a bit of insight into the challenges that 
a clustering algorithm must face in different situations. More important thing to note 
from the above discussion is that, a single clustering algorithm is not desired to work 
in all scenarios of big data. 

Big data clustering principles thus broadly are one of the following three types: (1) 
Localized algorithm, (2) Subsampling-based algorithm and (3) Dimension Reduc- 
tion (and Localized algorithm). In Sect. 4, we will discuss a few subsampling-based 
and dimension reduction-based clustering algorithms that exist in the literature. 
Unfortunately the principle of localized algorithm has failed to receive the desired 
attention. At this point we prefer first to elaborate more on this principle. 


Localized Clustering Principle—As transpired from our discussion, localized algo- 
rithms are absolutely required when our big data problem in turn imposes data immo- 
bility issue. Any such algorithm would require a two step procedure. In the first step, 
we would extract clusters from a local data storage. Velocity in the data, say, data 
collected over ‘days’, then requires that this first step procedure be carried out in 
sequential manner. The second step which is more algorithmically challenging is to 
identify which clusters in the current ‘day’ are new ones and which ones are same 
as some of the clusters from previous ‘days’. This step requires checking ‘equality’ 
of two clusters. Then we need to answer the following two questions: (1) what is the 
right measure to check for this ‘equality’, and (2) how do we implement this checking 
computationally efficiently and without moving too much of data around? In Sect. 5 
we shall present a two-step version of good old K-means algorithm. This two-step 
procedure impressively cuts down the cost of computation and memory requirement. 
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Recall that in high dimensional data clustering of variables we are required to pro- 
duce relevant clusters, e.g. most significant genes for a microarray gene expression 
data. This added requirement makes even basic clustering algorithms quite compu- 
tationally expensive. Thus, in such a scenario if we adopt the two-step principle dis- 
cussed above over a sampling-based algorithm, we might be quite efficient in achiev- 
ing our goal. In Sect. 5, we will present an algorithm that is based on our ongoing 
work following this principle in a rigorous manner. 

In the next section, we shall discuss the clustering algorithms available in the 
literature which have the hope to be useful in big data analytics. 


4 Literature Review 


The literature of clustering is flooded with many innovative methods that are applica- 
ble to data under several situations. Among them K-means and hierarchical cluster- 
ing are very old and stably used in clustering of different types of data. We must note 
that here we only consider clustering of continuous data vectors and not other types 
of data that include categorical data. Computational efficiency is one of the most 
important features and to this end we find other clustering algorithms that mainly 
emphasize this aspect of clustering. However, at the end, cluster quality is the main 
issue that we should concentrate as the next level analysis solely depends on this 
quality. But the main question is whether these clustering algorithms can be applied 
with high degree of efficiency to the big data as well. Brief reviews of big data clus- 
tering by [3, 16] are worth mentioning in this context. They have correctly identified 
the difficulties and challenges for applying standard clustering algorithms to big data. 
Due to its sheer volume only, big data can make our lives difficult because the com- 
putational time is too high and we might not get reasonable clusters at all using a 
single computer. Moreover, even with a high end computer, the computation may be 
slowed down severely due to the tremendous volume of data that is usually kept in 
the RAM of the machine. 

Thus high complexity and computational cost sometimes make the clustering of 
big data prohibitive. Even for K-means clustering, this creates problems when the 
number of data vectors is not so large compared to the dimension of each data point. 
Here big data puts more importance to the dimension rather than the size of data set. 
Hence in such situations scalability is the main challenge. Moreover some adaptation 
of these standard algorithms sometimes might cause substantial sacrifice in terms of 
cluster quality. Although multiple machine-based techniques can be proposed, these 
should not be compromised with the quality of clustering and should satisfy some 
optimum properties. 

Unlike partitioning around memoids (PAM) [11], clustering large applications 
(CLARA) [11] decreases overall time complexity from O(n’) to a linear order. 
Moreover clustering large applications based on random sampling (CLARANS) 
[15] method is proposed to improve the efficiency over CLARA. Another method 
BIRCH [20] takes care of clustering of large data sets that are even larger than the 
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memory size of the machine. Some density-based methods are available in the liter- 
ature that search for connected dense regions in the sample space and identify areas 
that are heavily populated with data. Such methods viz, DBCURE [12], DBSCAN 
[4], Optics [2, 14], DENCLUE [6], etc. are mainly used to construct clusters with 
irregular shapes but are not suitable for high dimensional data set. 

Hierarchical clustering mainly is of two types: divisive and agglomerative. Hier- 
archical divisive methods start with one single cluster and try to divide it into other 
clusters until each point is put in a different cluster. Hierarchical agglomerative meth- 
ods work exactly in the opposite manner but the philosophy of considering each point 
as one cluster remains the same. Here it starts with each point and try to merge other 
points into a bigger cluster. Both these approaches use different types of distance 
measure between groups. Naturally hierarchical clustering have complexity as high 
as of order n* and hence not scalable, making it not at all appealing for big data 
clustering. 

However, in big data clustering perspective, some proposals have been put for- 
ward using the computing power of a single machine and using multiple such 
machines. Parallel clustering and MapReduce-based clustering are the most popular 
ones. DBDC [1, 8] and ParMETIS [9] are some examples of multilevel partitioning 
algorithms that improve DBSCAN and METTIS, respectively, for parallel computa- 
tion. MapReduce is a framework that is initially represented by Google and its open 
source version is known as Hadoop. MapReduce-based K-means clustering, known 
as PKmeans [21], is an adaptation of standard K-means algorithm that distributes 
the computation among multiple machines using MapReduce. The main objective 
of PKmeans is to speed-up and scale-up the computing process. Another version 
of K-means is ELM K-means [19] that is seen to be suited nicely for finding good 
quality clusters on less computation time thus showing a promise to its adaptability 
to big data clustering. Similarly MR-DBSCAN [5] is a scalable MapReduce-based 
version of DBSCAN that can be used in multiple machines. 

Although clustering algorithm when applied in parallel, increases the scalability 
and speed, evaluation of such an application is needed for understanding its use- 
fulness as compared to the application of a single clustering to the whole data, if 
possible. Thus results that come from different machines should be evaluated to see 
how the actual clusters look like; whether they really form “good” clusters in terms 
of distance-based measure between clusters and within clusters. With technological 
advances speeding up of computation would be more feasible and possible in future 
to a great extent, but statistical validation of the quality of clusters thus produced 
should be kept in mind and needs to be studied elaborately. Nowadays GPUs are 
much more powerful than CPUs and so focus is on developing appropriate algo- 
rithms and apply them in a distributed manner to increase the scalability and speed. 
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5 Clustering Meta Algorithm/Partition-Repetition 
Approach 


We dedicate this section to present two algorithms for big data clustering. These two 
algorithms are motivated by our discussion in Sect. 3. However, our approaches can 
be extended easily to incorporate any other appropriate clustering engines. These 
proposal for generic algorithms are based on the need of analysis and nature of big 
data. It can be adapted for specific purpose with appropriate adjustments. Our pro- 
posed method is based on the philosophy which we call Partition-Repetition algo- 
rithm, as explained below. 


Partitioning—Suppose that a dataset consists of n data points (e.g. genes). We ran- 
domly partition n points into L(n) parts. If we can run a clustering algorithm on a 
data of size M, we partition the data into L(n) ~ al parts. To reduce the error due 
to partitioning one may wish to repeat this procedure R(n) times so that we are left 
with Mọ = R(n) x L(n) manageable datasets and apply the clustering algorithm of 
our choice on each such dataset. We store all clusters from Mọ datasets in an empty 
bucket 4. The elements in this bucket can be enumerated as 


B = {C,,Cy,...,Cy, }. 


Combining—Consider the first member of the bucket C,. We want to combine all 
members of the bucket with the similar pattern with that of C, and redefine it as 
a new collection. Let us consider a decision function z(-,-) that maps 2* x 2* to 
{0, 1}, where 2* denotes the power set of X. If z(C, C ) = 1, then we decide whether 
the subsets C and C are of similar profile or they are of different profiles. With the 
help of this decision function, bucket elements are redefined as, 


Mo 
C=C | (Ss (Cpr (c6) (1) 


and 

C; := I (Gl (C C;)) forj=2,3,..., Mo. (2) 
Here, J (+, -) is a set valued function defined on 2* x {0, 1} as follows: 2 (C, 1) = C 
and .4(C,0) = @. At the end of this redefinition step, we are again left with Mọ 
members in the bucket but now possibly some of them are empty. This process of 
redefining is carried out sequentially for i = 2, 3, ... , Mọ — 1 as follows, 


Mo 
GC) (uy (CC sc )) (3) 
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and 
C; := I (C, 1-a (CC )) forj=itl,...,Mo. (4) 


A necessary constraint on the decision function z is that z(¢, C) = z(C, d) = 0 for 
any C C X. 

We can denote this method as an operation on %, say 2 := CCF). In a sense of 
generality, the above method, specified by the decision function z, is one of many 
possible methods that operates on ¥ and redefines its members as having required 
characteristics. 

The two clustering algorithms discussed below basically exploit these two ideas 
of partitioning and repetitions. Depending on the exact algorithm, there might be 
some additional work; however, the basic principle always remains the same. 


5.1 Adaptive K-Means Clustering for Big Data 


We focus on big data with fixed dimension and huge number of individuals with 
velocity. Then, our discussion in Sect. 3 guides us to use a localized algorithm in 
order to cluster the dataset. We need to accommodate for noise in the data, growing 
number of clusters and other finer details for our algorithm to work. Rather than being 
explicit about the algorithm, we are going to present here a comprehensive sketch 
of what we call an adaptive K-means clustering for big data with fixed dimension. 
Tackling of noise is not considered here; we will discuss it in more details later. 

Standard algorithmic practice for efficient algorithm in big data analytics is to 
work with pointers or indices to individual data points and process only a small 
number of indices at a time. This practice can be implemented quite easily for 
our proposed adaptive K-means algorithm. Let us consider a data stream of sizes 
Ni, N3, ..., Np, ... 1.€. on ‘day?’ 5 we have data of size N;. Suppose that at a time we 
can only accommodate around M data points for K-means clustering. 


Partitioning—Partition each stream of the data so that each partition contains at 
most M data points. Here of course by partitioning the data we mean partition- 
ing the indices; there is no need to consider the data yet. Let these partitions be 
(Pips PiL, hres i ere Por}, ..., where {P.,,... Por } is the partition of s-th 
data stream of size N,. 

Apply K-means algorithm on each of these partitions to get clusters (Cc. ws Cl 
from P,,. Again it is only required to store the grouping of pointers of the clustering 
for further processing. 


Combining—Start with the first set of clusters {C fe sr G a }. Consider the next set 
{C A eg C}. Use sequential comparing and combining operation similar to that 
discussed earlier in the section to identify distinct collection of clusters. For each 
J =1,2,...,k check if C, is “equal” to C for any i. This equality of two clusters 
can be conceived in many ways. One way is to check whether the two clusters are 


coming from the same probability distribution as a result of random sampling. One 
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can also think whether the distance between these two clusters is so small that they 
can be taken as one single cluster. This checking of equality of two clusters can be 
defined by the user and implemented as a testing method accordingly. If there is one 
such i for aj then relabel the indices of C , of being in C R If there is no such i then 


C, is a new cluster. 

We have mentioned in our earlier discussion that this operation is expensive both 
in terms of memory and computation. We can avoid this expense by applying the 
checking procedure of two clusters C and C’ based on a representative subsample. 
Thus we can also follow our localized algorithm principle in order to save memory 
space and speed up the process. 

The nicest thing of this very simple looking algorithm is that it allows a sequential 
processing. A data stream collected at high frequency can be very well adapted to 
this algorithm. The usual concern with choice of k is secondary here and can be 
quite easily accounted for using domain knowledge, e.g. if partitioning in the data 
stream is as per intervals of time then domain knowledge can let us know many 
similar groups of pattern we can expect in that interval of time. Consequently the 
combining method allows us to discover new clusters over time. 

In this description of the algorithm we have deliberately ignored the possible 
noise in the data. To practically address this problem we need to put the K-means 
algorithm inside a wrapper that filters the noise along with extracting the clusters. An 
extensive discussion to address this issue is discussed in the next subsection where 
we have also proposed an algorithm that directly tackles the noise points and pick 
clusters at the same time. 

Equality of two clusters can be checked in many ways. One can use any non- 
parametric multivariate test for equality of two distributions. However, usually they 
are not so powerful. We can use spanning tree and check the span of each cluster 
compared to the combined one. Then based on a certain criterion, we can decide 
upon whether two clusters are same. Another option is to use concentration ellipsoid 
and check whether it contains at least 90 % of points within the ellipsoid of the com- 
bined clustered data points. Some other adequate methods can also be thought of to 
settle this issue. The choice of test or logic behind combining two clusters might also 
depend on the objective that underlies the clustering of the data set. 


5.2 Adaptation of Tight Clustering to Microarray Data 


Microarray gene expression data is a type of data with high dimension but for consid- 
erably small number of individuals. We are interested in extracting prominent gene 
clusters which would shed light on finding co-regulation and specific pathways. This 
type of data is quite natural and common in big data analytics domain. 

But as we have discussed earlier, there is a very challenging goal to be achieved 
while clustering such data. We are looking for clusters of genes which might be of 
tiny sizes (e.g. 10—50 genes out of hundreds of thousands) and they are very ‘tightly’ 
clustered. Thus most of the variables are present as unnecessary noise. Tseng and 
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Wong [18] proposed a detailed and carefully constructed algorithm, named ‘tight 
clustering’, to achieve this goal. Given a microarray data matrix of n genes, tight 
clustering algorithm runs K-means as a sub-algorithm based on a number of con- 
secutive choices of k. For each of these different choices of k, it chooses the possible 
candidates for the tight-most cluster based on K-means algorithm applied on a num- 
ber of subsampling of n genes. With these choices of tight-most cluster at hand, in the 
next step the algorithm picks the stable one, thus extracting the first stable tight set 
of genes that are likely to belong to a functional pathway or having some similarity. 
The second iteration for extracting the next set of similar gene patterns repeats the 
method on the remaining set of genes. Among several nice benefits of tight clustering 
a few are worth mentioning, 


e Tight clustering method reduces the local minimum problem of K-means cluster- 
ing to a certain extent. 

e The algorithm always ensures stable clusters and extracted set of patterns that are 
ordered in accordance with their tightness. 

e For the resulted clusters C,’s we have the relaxed constraint > #C,<n.Asa 
result the choice of number of similar gene expression pattern becomes irrelevant. 


The drawback of this algorithm is that it is computationally very demanding. To 
pick the tight-most and stable cluster it requires a clustering algorithm to be car- 
ried out a considerable number of times. While with a little intelligent implemen- 
tation of the algorithm one can avoid falling into the trouble of memory problem, 
computational expense have to be paid. Our principle in Sect. 3 guides us to use a 
subsampling-based approach. Such a method would sample a portion of genes and 
cluster these subsamples and then output a cluster based on the clusters on the sub- 
samples. But we have to keep in mind the target of extracting tightly clustered genes 
whose size might be very small. If we rely on a subsampling-based method, we would 
most certainly fail to reach that target. This is where we can take advantage of our 
partition-repetition principle. 


5.2.1 Tight Clustering for Huge Number of Genes 


Tight-most Cluster—Applying partition-repetition process, we get a bucket & 
where members of the bucket are candidates for the tight-most cluster from the whole 
data. To achieve this, on each partition of the data we apply tight clustering algorithm 
and collect the very first tight cluster and process this collection through the com- 
bining process. In order to obtain the tight-most cluster from the possible choices, 
i.e. members of Z, we require a measure of tightness of a set of data points. Suppose 
o(C) is a positive valued function defined on non-null subsets of X that tells us how 
disperse the set of data points C is. Then our desired cluster will be given by, 


a). 
Ci Ane Ma (5) 
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The stability of this cluster is ensured as each output of the tight clustering method 
ensures a stable cluster. 


Next Iteration—After we have obtained the tight-most cluster from the data, to get 
the next one in line, we remove all the data points of the obtained cluster from the 
whole data as well as from members of the bucket. Thus we are left with the pair 
(X\ C, B := {B\ CY|B € }). 

Now the remaining data points are partitioned and tight-most cluster of each of 
these parts is put into the bucket %. Notice that, unlike last time, we possibly already 
have a non-empty collections in the bucket. The operation C is carried out on the new 
bucket and just like the last time and the next in line tight-most cluster C® is obtained 
by comparing dispersion measure of the members of Z. 

Continuing this way we extract k clusters, as (C OPO) adC ©) of sizes, respec- 
tively, n4, n2, ... , ng (say). The remaining data points, that do not belong to any clus- 
ter, are then identified as noise. The ability of filtering out the noise as a by product 
is a wonderful benefit of this iterative clustering algorithm. 

In our discussion here, rather than providing the details about the algorithm and 
the analysis, we have focused on presenting the idea behind the implementation of 
the clustering that is adapted to big data. The adaptation is appropriate and justified 
mainly because partitioning and then the combining procedure ensures the quality 
of the cluster thus obtained. 


6 Simulation Studies 


In this section, our proposed algorithms are being demonstrated through two types of 
clustering as discussed in the last section. One is a version of K-means that is adapted 
to cluster big data. This method provides clusters from different segments/‘days’ of 
the whole data and attach cluster level to each data point. On the other hand, the 
other type of clustering that we discussed is an adaptation of tight clustering method. 
This method distributes a part of the whole dataset, in most cases a major part, to 
different clusters and then leave some data points as noise points. These noise points 
are interpreted as having entirely different properties that do not allow them to go 
into any of the clusters thus formed. 

To evaluate these algorithms, we have done extensive simulations by generat- 
ing different types of datasets with size of each data being very large. Our simu- 
lation shows that the algorithms hold good in principle nicely and can be applied 
to any real dataset that is huge in volume. In order to demonstrate the performance 
of adaptive K-means algorithm, we generate datasets of size 1000000 (1 Million) 
with varying dimension (p), i.e. for p = 4,6, 8, 10, and 20. In each case we gener- 
ate the data from five multivariate normal distributions so that it should have five 
clusters. Mean of each variable is taken to be 5(i — 1) where i denotes the clus- 
ter level (i = 1,...,5). The standard deviation is always equal to 1 and pairwise 
correlation coefficient between two variables is taken to be 0.6. Thus if a cluster 
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Fig. 1 Rand index (with 1 standard deviation margin for 10 runs) of adaptive K-means algorithm 
over varied dimension 


level is 2, then each data point in that cluster is a random vector observation from 
N,,G1,, 0.6/, + (1 — 0.6)11’) where p is the dimension of the vector. The positive 
dependence between the variables ensures a small amount of overlapping between 
the clusters. To analyse the performance of the algorithm, we input the value of k as 
5 and get the corresponding clusters after running our proposed adaptive K-means 
algorithm. In each case, since we know the true clusters, we can calculate Rand index 
to evaluate the performance of the algorithm. Higher the Rand index, better would 
be the performance of the clustering algorithm. In Fig. 1, we plot Rand index for 
the respective datasets for different dimensions. This is a good way to look how effi- 
ciently the algorithm works for varying dimension. The plot shows that the algorithm 
performs very satisfactorily in this set up. It is important to analyse the performance 
over varied dimension since random partitioning in the adaptive K-means algorithm 
could have had adverse effect on the quality of the clusters. The plot shows that this is 
not the case. Further, the run time of the algorithm is merely a couple of minutes on 
a non-parallelized implementation on a standard machine. The plotted figure is not 
chosen with any bias; very similar plots are seen for many other simulation scenarios 
with different k; k being varied over ‘days’ and different types of data clouds. 
Generation of data that can be applied to see the efficiency of adaptive K-means 
algorithm is different than if we want to assess the performance of our modification 
of tight clustering approach. The main reason is that tight clustering method should 
be applied in some special situations where the general pattern of the data does not 
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allow all points to go into one cluster or other. Rather, in such cases, there is a pro- 
vision of a data point to remain alone in the data cloud and may be termed as noise. 
Microarray gene expression data is a nice example of one such situation. Keeping 
this in view, we have generated data that resemble microarray data. Based on a hier- 
archical lognormal distribution we generate this data [17] of size 100000 (=10°) and 
for varying dimension as 5, 10, and 20. In each case we consider again only five clus- 
ters but vary percentage of noise as 5, 10, 15, and 20 %. Here again, since we know 
true clusters, we can calculate Rand index for each clustering and can check the effi- 
ciency of the algorithm. Moreover, plotting Rand index against percentage of noise 
reflects the effect of the amount of noise in the data. Also a plot against dimension 
would give us an idea how the performance of the algorithm changes over dimen- 
sion. We run tight cluster algorithm specifically modified to adapt to big data for 10 
times for a single data set and calculate the mean Rand index along with its standard 
deviation. We plot in Fig. 2, the mean Rand index against percentage of noise with 
error bars being equal to the standard deviation for each dimension. In Fig. 3, we plot 
Rand index against sample size across noise percentage. Finally in Fig. 4 we look at 
the performance as we increase dimension with varying noise level. These graphs 
clearly explores the performance of our algorithm that leaves a portion of data as 
noise. 
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Fig. 2 Rand Index (with 1 standard deviation margin for 10 runs) of proposed tight clustering 
algorithm across dimension and varied noise level 
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Fig. 3 Rand Index of proposed tight clustering algorithm across sample sizes and varied noise 
level 





oO 
O) 5% noise 
o , 
5 10% noise 
oO 15% noise 
O 20% noise 
O 
O 
b 
LO 
oO œ 
N f 
— O 
” 
ca) 
oD. 
= 
D 
= © 
= oO 
x< 
(€b) 
O 
„= 
E 
GR 
So 
oS 
= 
Oo 
5 10 15 20 
dimension 


Fig.4 Rand Index of proposed tight clustering algorithm across dimension and varied noise level 
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7 Concluding Remarks 


Huge volume, diverse variety and high velocity sometimes make the standard statis- 
tical tools challenging in the domain of big data analytics. Clustering is definitely one 
such area where some modification of the existing algorithms as well as the adapta- 
tion and implementation of techniques to big data is required. Clustering algorithms 
demand computational efficiency along with scientifically reasonable arguments that 
ensure picking good quality clusters from the data cloud. Naturally this faces con- 
siderable challenges when the data volume is too large and grow with high velocity. 
Current state of the art does not discuss this issue in a rigorous way and is little scat- 
tered. Existing literature mainly evaluates the existing clustering algorithms to its 
adaptability to big data domain. But focussed research is needed towards resolving 
this challenge. 

To this end, we have proposed two algorithms that are adapted specifically to 
tackle the big data issues in order to find good quality clusters that are amenable 
to further statistical analysis. These two algorithms are suited for two separate big 
data scenarios. First one of them is an adaptive version of K-means algorithm that 
is proposed for data with very high velocity but fixed dimension. The later is a scal- 
able version of tight clustering algorithm that is proposed for high dimensional gene 
expression data or medical imaging data or any other data having this feature. 

Adaptive K-means algorithm is very fast and cluster data points efficiently. This 
method as we proposed can be used in a sequential manner and combined with the 
previously obtained clusters as and when required. It can be modified and applied 
to a portion of data instead of the whole data and can easily ignore data beyond a 
certain time in the past if required by the situation. Naturally this flexibility is quite 
appealing to tackle the problems due high velocity of data flow. As apparent from 
Rand index values in our simulation study, our adaptation also ensures good quality 
clusters, even after partition of relevant or available portion of dataset, that is usually 
very large. 

Tight clustering method specifically adapted to big data clustering in presence of 
noise is an interesting one to identify scatter points even with considerable veloc- 
ity of data coming into the domain of study. Noise points may arise due to many 
reasons, an important one being the diverse variety of data. Since partitioning is 
an inherent feature of our proposed adaptation of tight cluster algorithm, it auto- 
matically takes care of the ‘variety’ characteristic of the data. Although little more 
computation intensive, this algorithm is able to tackle the volume of big data in an 
efficient manner as transpired from Rand index values in our simulation study and 
is quite effective over dimensions and increasing percentage of noise. 

Idea of partitioning is instrumental in tackling the large volume of data, but com- 
bination wrapper inside our algorithm ensures the quality and quantity of clusters 
thus obtained. Our objective is not just to apply clustering algorithm to a partition of 
a dataset, but the main issue is to find clusters that can be obtained if there is hypo- 
thetically huge space to consider the whole data together. Hence combining different 
similar type clusters is the key to produce the end product and hence should be done 


92 B. Karmakar and I. Mukhopadhayay 


efficiently and quickly without losing mathematical and statistical rigour. Our pro- 
posed wrapper is able to find reasonable number of clusters with very good quality 
making a sense of the data cloud as a whole. 

We have written R code to implement these algorithms. They can be easily made 
parallel if one wants to do the computation in a computing cluster consisting of many 
nodes or in different machines. Even the data storage can be done in separate boxes or 
memories so as to apply the algorithm. Although we describe here only two methods, 
the basic idea can be applied to any standard clustering method thus enabling us to 
produce good quality clusters from a real big data. Many functions that are wrapped 
inside the algorithm can also be chosen by the user depending on the real need of 
the problem. 

As a future research direction we want to make a note of the fact that, in terms of 
clustering algorithms an open field remains where velocity brings with it increasing 
number of features for individuals. In Sect. 3 we have provided a peek into possible 
algorithmic principles for such data. A practical method for clustering such data 
would be beneficial to large consumer facing industries. 
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Online Graph Partitioning with an Affine 
Message Combining Cost Function 


Xiang Chen and Jun Huan 


Abstract Graph partitioning is a key step in developing scalable data mining 
algorithms on massive graph data such as web graphs and social networks. Graph 
partitioning is often formalized as an optimization problem where we assign graph 
vertices to computing nodes with the objection to both minimize the communica- 
tion cost between computing nodes and to balance the load of computing nodes. 
Such optimization was specified using a cost function to measure the quality of 
graph partition. Current graph systems such as Pregel, Graphlab take graph cut, 1.e. 
counting the number of edges that cross different partitions, as the cost function of 
graph partition. We argue that graph cut ignores many characteristics of modern 
computing cluster and to develop better graph partitioning algorithm we should 
revise the cost function. In particular we believe that message combing, a new 
technique that was recently developed in order to minimize communication of 
computing nodes, should be considered in designing new cost functions for graph 
partitioning. In this paper, we propose a new cost function for graph partitioning 
which considers message combining. In this new cost function, we consider 
communication cost from three different sources: (1) two computing nodes establish 
a message channel between them; (2) a process creates a message utilize the 
channel and (3) the length of the message. Based on this cost function, we develop 
several heuristics for large graph partitioning. We have performed comprehensive 
experiments using real-world graphs. Our results demonstrate that our algorithms 
yield significant performance improvements over state-of-the-art approaches. The 
new cost function developed in this paper should help design new graph partition 
algorithms for better graph system performance. 
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1 Introduction 


Graph-structured data are playing a central role in data mining and machine 
learning. Such data are widely used in application domains where linkages among 
objects are as important as the objects. Examples include the web, social networks, 
semantic web, recommender systems and biological networks among others. 

Large-scale graph data ask for scalable systems that can scale to graphs with 
millions or even billions of nodes. There are several systems supporting intensive 
computations on input graph data. Examples include Pregel [1], Giraph [2] and 
Spark [3]. There are also systems support iterative machine learning computations 
such as approximate Bayesian inference in systems such as Graphlab [4], or for 
efficiently resolving queries in large-scale databases Neo4j and Trinity [5]. 

In designing systems supporting large-scale graph mining and learning systems a 
key question is graph partitioning. In graph partitioning we map vertices of a graph 
to computing nodes in a cluster so that the overall computation efficiency is opti- 
mized. In such mapping we considered two critical factors: communication cost and 
load balancing. Since we map vertices to different nodes, there is a natural problem 
of inter-node communication cost where we send a message from a node to another 
one for information exchange or computing synchronization. Since computing on 
graphs is typically iterative and different vertices may have different amounts of 
computing, in load balancing, we would like to have the state where each com- 
puting node has similar work load. As studied before graph partitioning often 
affects the computational performance of any graph mining and learning systems 
dramatically [6]. Many heuristic and approximation algorithms were proposed to 
solve the related optimization problems in graph partitioning [7—9]. 

Reducing communication cost is one of the most important problems in graph 
partitioning and how to devise the cost function is the key issue. Several popular 
graph computing algorithms, including Pregel and Graphlab, take graph cut, 1.e. 
counting the number of edges that cross different partitions, as the cost function of 
graph partition. Graph cut is a simplified abstraction of the communication cost and 
it ignores many characteristics of modern computing clusters. For example, the 
communication cost on creating a new message channel between the different 
partitions is far greater than exchanging a short message between the two machines. 

To design better graph partitioning algorithms, we argue that different communi- 
cation patterns should be considered. Formally, we proposed a new cost function for 
graph partitioning including message combining. In the new cost function, we con- 
sider communication cost from three different sources: (1) the cost that two machines 
establish a message channel between them; (2) the cost that a process creates a 
message utilizing the channel and (3) the cost that a process sending a message 
through the established communication channel. Based on this cost function, we 
developed several heuristics for large graph partitioning. We call this function affine 
cost function since we use a weighted linear combination of the three factors. Com- 
prehensive empirical experiments on real-world graphs demonstrate that our algo- 
rithms yield significant performance improvements over state-of-the-art approaches. 
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The rest of the paper is structured as follows. In Sect. 2 we survey current graph 
partitioning algorithms, grouped by the heuristic that were used. In Sect. 3 we 
introduce the background of current cost function and three heuristic algorithms. In 
Sect. 4 we define affine graph partitioning cost function. In the same section we 
introduce two online heuristic algorithms solving the related optimization problems. 
We present the empirical evaluation of the new algorithm together with those from 
state-of-the-art heuristic algorithms in Sect. 6. Conclusion and discussion are 
described in Sect. 7. 


2 Related Work 


Graph partitioning is often an NP-hard problem and solutions are generally derived 
using approximation and heuristic algorithms. 

Traditional approximate solution focuses on the (k, v)-balanced partitioning 
problem. In the (k, v)-balanced partitioning problem we divide the graph of n ver- 
tices into k partitions; meanwhile, the number of the vertices of each partition is not 
greater than v * n/k. In the special case of k = 2, the problem is called the Minimum 
Bisection problem. Several approximation algorithms exist for the minimum 
bisection problem [7, 9]. Some of those algorithms can be extended to larger values 
of k. The algorithm by Saran and Vazirani gives an upper bound of n * (k — 1)/k 
for the (k, 1)-balanced partitioning problem. Note however that this algorithm 
requires a running time that is exponential in k. A critical limitation of approxi- 
mation graph partitioning algorithms is that they often run in quadratic time. For 
graphs with millions or billions of nodes, linear or sublinear running time is pre- 
ferred and quadratic processing time is prohibitive. 

Heuristic algorithms are widely used in graph partitioning as well. We categorize 
such problems into two groups: offline graph partitioning and online graph 
partition. 

A common offline approach is to construct a balanced k-way cut in which 
sub-graphs are balanced over machines and communications between machines are 
minimized. Offline methods, such as Spectral Clustering [10], Metis [11] and K- 
partitioning [12], collect full graph information to perform offline partitioning. For 
example, widely used algorithm Metis uses a heuristic algorithm that iteratively 
reduces graph sizes with multiple stages. Once we reach a workable size, we start 
the partitioning phase from the smallest graph and refine the partition from smaller 
graphs to larger graphs. Metis usually achieves good cuts but exhibit poor scaling 
due to high computation and memory costs [13]. Especially, these algorithms 
perform poorly on power-law graphs and are difficult to parallelize due to frequent 
coordination of global graph information. 

In online graph partitioning vertices (or edges) arrive as a steam. We place each 
vertex (edge) using a “on-the-fly” approach in order to minimize the overhead of 
computing. Two commonly used classes of such algorithms are those based on the 
hash or greedy heuristic. 
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Hash or random partition is the simplest method. It supports both vertex and 
edge partitions. Random partition of vertices is first initiated and is popular for 
many graph processing platforms such as Pregel [14], and related algorithms such 
as Apache Giraph, PEGASUS [15] and GraphLab. Vertex random partition assigns 
graph vertices to one of the k machines uniformly at random. This heuristic method 
is efficient for balancing the number of vertices over different clusters but ignores 
entirely the graph structure and may not perform well on minimizing the com- 
munication cost. To cope with the poor performance of vertex random partition on 
real-world power-law graphs, such as social networks and the web, Powergraph 
[16] proposed random edge partition which assigns each edge rather vertex to one 
of the k partitions uniformly at random and employ a “master-mirror” technique to 
keep the consistent of the nodes “split” in different clusters. Theoretical and 
empirical studies have demonstrated the advantage of edge partition over vertex 
partition [16, 17]. 

Greedy algorithms improve the random algorithm by introducing heuristics to 
vertex or edge assignments. The recent greedy algorithms introduced and discussed 
by Stanton and Kliot [18], as well as the latest Fennel method [19], usually consider 
some greedy rules in graph partition. For example, the DG (Deterministic Greedy) 
and LDG (Linear Weighted Deterministic Greedy) [18] algorithms usually assign a 
vertex (edge) to the partition that holds the largest number of its neighbours. 

From the cost function point of view current approximation algorithm or 
heuristic algorithm only consider the number of edge cut in computing the com- 
munication or message cost. Such simplified abstract has the potential to be 
improved by incorporating the different kinds of message costs which are closer to 
real cluster systems. 


3 Background 


In this section, we introduce notations that we use throughout this paper. 

A directed graph is a tuple where G = (V, E) with the set of vertices V and the set 
of directed edges ECV XV. n =IVI and m =IEI are number of vertices and the 
number of directed edges, respectively. Let Nin(u)={vEV,(v,u) EE} and 
Nou(u) = {v EV, (u,v) E E} denote the set of vertices that have edges, respectively, 
coming into and leaving away from u E€ V. di, (u) and dout(u) are the in-degree and 
out-degree of vertex u € V, i.e. din(u) =|Nin(u)| and dour(u) = |Nour(u)|. Note that 

© din(u)= È, dou(u). Let S; be the set of vertices in partition i (S;CV). Let N 


uceV ucV 
(v) be the set of vertices that are neighbours of vertex v. That is to say the vertex in 


N(v) is incident to or from vertex v. In our context, a partition is the set of vertices 
that are assigned to the same machine (computing node) in a computer cluster. In 
this sense, the two terms, a partition and a machine, are interchangeable. 

We use a |V| xk binary matrix X to represent a partition schema of a graph with 
n =IVI vertices into k machines. We define X,, ; = 1 if vertex u € V is assigned to a 
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machine i€|1...k], and X,,;=0 otherwise. Each vertex is assigned to exactly one 


machine and for any valid graph partitioning schema X we have a ı Xu,i= 1 for 
all uE V(G). 

Each graph partitioning problem is characterized by a communication cost 
(denoted by C) which has to be minimized under a condition of balanced loads of 


the clusters. If we denote L;(X)= )}) X,; the load of machines i € [1...k], i.e. the 
uEeV 


set of graph vertices that are assigned to the machine 1 for a schema X, the graph 
partitioning problem can be written as 


Minimize: C(X) 41) 
. : Die n Li (X) 
Subject to: Max; ej. Li(X) <42 
Here, v is an off-balancing coefficient. 
A way to define the cost function C used in previous equation is to count the 
number of edges cross different machines. For a cluster with k machine, this cost 
function is formulated by 


C(X)= YE Kall Xi) 2) 


(u,v) EE 


This cost function is used by many current heuristic algorithms. There are three 
widely used heuristic online algorithms. They are compared with our algorithms in 
Sect. 5. 


e Random/Hash Partition: 
S; chose uniformly by random. Assign v to partition i, we use i = (random 
number mod k) + 1; 

e Deterministic Greedy (DG): 
Maximum |N(v) nS;| 

e Linear Weighted Deterministic Greedy (LDG): Maximum 
IN(v) ASIC — |SiAZ)) 


The random partition heuristic is the simplest method to partition the graph and 
is widely used. Deterministic Greedy algorithm (DG) considers reducing the edge 
cut by assigning the vertex to the partition including most number of v’s neigh- 
bours. Based on DG, Linear Deterministic Weighted Greedy algorithm (LDG) uses 
an item (J — |SilXn/k)) to balance the partition. 


3.1 Cost Function with Message Combining 


In the previous section, the communication cost is simply the number of edges that 
cross different partitions. This may not reflect a realistic situation. Pregel proposes a 
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Fig. 1 Vertex partition with 
combining 





method called message combining and allows users to combine messages that are 

sent from a partition to the same vertex v in another partition. These messages are 

first combined into a single message and then sent to the vertex v in another 

partition. Similarly, the message sent from one vertex v in a partition to the different 

vertexes in another partition can also be combined into a single message (Fig. 1). 
Formally, the cost function (C”*) with message combining is [17] 


(1- lI 1-x,)) + (1- I] 1-9) 
v ENin(u) w E Nout(u) 


k 


C™(X)= È È Xu È 


uEVi=1 ji 








(3) 


Current heuristic graph partitioning algorithms do not consider combining in 
graph partitioning. The common method to handle combining is to first partition the 
graph and then use combining strategy to reduce the message or edge cut. 


4 Methodology 


4.1 Cost Function with Affine Message Combining 


In this section we consider a more realistic cost function that can be applied in 
modern cluster computing environment. 
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We notice that in current clusters sending a message from one partition to 
another partition may have different costs depending on several factors. Creating a 
new message channel between two different partitions that have not message 
channel between them will incur overhead larger than sending a new message 
between two partitions that have already a message channel. In addition, if two 
messages between two partitions can be combined to one message, the commu- 
nication overhead incurred by such a combined message is less than that of two 
messages that cannot be combined. Based on this consideration we argue that 
different types of graph cut and combining issue should be considered in designing 
optimal cost functions for graph partitioning. 

We use an affine function to formalize a cost function that we believe better 
characterize communication cost than graph cut, as shown below: 


C(x) =w, Xn, +W2 Xm +W3 XN3 (4) 


In this cost function w; denotes the cost of creating a new message channel 
between two different partitions. wz denotes the cost of creating a message utilizing 
the channel and w3; is the cost of sending a message with one unit length. Here the 
length of the message means the number of messages that have been combined to 
one message on condition the combining method. In real clusters w; is far greater 
than w> and w> is greater than w3. n;, n2 and n3 are numbers of those three different 
messages (edges) in the graph. Figure 2 is an example to illustrate the cost of 
different types of messages. 

We illustrate the new cost function in Fig. 2. There are message channels between 
partition M; and M, as well as M, and M3 so n; is 2; there exists two messages utilized 
the message channel between partition M; and M3: one is the combined message 
(V,;Vs combined with V2V5) and the other is a single message (VsV3). Their message 
lengths are two and one separately. Therefore, n> is 2 and n3 is 3 for messages 
between partition M; and M2. Similarly, there exists one message (n2 = 1) utilized, 
the message channel between partition M and M3, and the length of such message is 
three (ViV7, V4Vs and V4Vo are combined to one message. Then here n3 = 3.). The 
overall cost of the graph partitioning illustrated in Fig. 21s: 2 xX w1 +3 X w2 + 6 X w3. 


Fig. 2 The different 
messages between partitions 
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According to the definition of n;, nz and n3 in (4) they can be formally for- 
mulized as 


n3 = 2 X Xui(d —X,;) 


(i,j) € [1..k] (u,v) EE 
The formal formula of the cost function with affine message combining is 


k 
Cum) =w] X > > (1 z I] (1 — XiX,j)) + Ww 
(i,j) €1 (u,v) EE 


k k 
x È È Xal- [[0-X%;) tmx È È Xull-Xy;) 
(i,j) € [1..k] u,v E V (i,j) € [1..k] (u,v) EE 


(5) 


5 Online Heuristics for Cost Function with Message 
Combining 


In this section, we present two online heuristics for graph partition. The first 
algorithm uses the cost function with message combining (Eq. 3) and the second 
algorithm use the cost function with affine message combining (Eq. 5). 

The online graph model has been described in previous work [18, 19]. We 
briefly outline the setup here. In this algorithm we have a computing cluster of 
k partitions, each with memory capacity À and the total capacity, Ak, is large enough 
to include the whole graph. The input graph may be either directed or undirected. 
Undirected graph can be taken as directed graph where each edge is bi-directional. 
As vertices arrive, our algorithm assigns the vertex to one of the k partitions. We 
never relocate a vertex once the initial assignment is made. 


5.1 Deterministic Combining Greedy Algorithm: A Message 
Heuristic for Online Graph Partitioning 


We propose a new heuristic online algorithm that considers the combining in the 
partition process. We call this algorithm Combining Deterministic Greedy (CDG) 
algorithm. It is an improvement on the LDG heuristic based on the cost function 
with message combining. 


Online Graph Partitioning with an Affine Message ... 103 


To clarify the score function used in CDG algorithm, we first define some 
formulation: 


N(v) ={wlwv € EorvwweE,weV} 
a. i) ={v|dw E Sp vw € Eorwv Ee EF} 
Si) =|N ai 

) = |N 

)=( 


i is ae 


9 


Si 
Si 
Si 
E = 


Here N(v) is defined as the neighbours of vertex v. In another word, it is the set 
of the vertices that are incident to or from vertex v. 

Similarly, N(S;) is defined as the neighbours of partition S;. In another word, it is 
the set of vertices that are incident to or from any vertex in partition S. 

The item |N (v) N S;| is also used by previous algorithms, such as DG and LDG. 
It computes the number intersection of vertex v’s neighbours with the vertices in 
partition S;. This item evaluates the message reduction by allocating vertex v with 
its neighbours in the same partition. 

The item |N(v) NN(S;)| is the number of intersection of vertex v’s neighbours 
with the neighbours of partition S;. This item evaluates the message reduction by 
combining the message incident to or from partition S;. 

H(S;) is used to balance the load of partitions. 

The score function used in CDG algorithm is 


Scag = |P(v, Si) +a xX A(v, S;)] x u(S;) (6) 


Since the Scpg 1s already has the item (S;) to balance the partition. The heuristic 
of CDG algorithm is simplified: 


Maximum: Scag 


This demonstrates that CDG not only tries to allocate the vertex v with its 
neighbours in the same partition but also tries to utilize the message combining and 
allocate v to the partition that more combining is possible. The parameter œ is the 
weight for combining item in the process of graph partition. 

We use a simple case (Fig. 3) to explain the graph partitioning process in CDG: 

When a new vertex Vg arrives, we need to consider three possible assigning, that 
is, casel for assigning Ve to partition Sı, case2 for assigning V6 to partition S and 
case3 for assigning Ve to partition S3. 

The previous heuristics (such as DG and LDG) will give case2 and case3 the 
same score because both Sz and S} include the same number of neighbours of Ve. 
However, CDG will give case 2 higher score because there exits another Vg s 
neighbour—V, in partition S; and at the same time V; is also the neighbour of the 
vertex V> in the partition S2 (V; E N(S;)). The edge V;V> and V;Vs has the same 
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Fig. 3 Assign a new vertex 
using online heuristic 





direction, which means if we assign Ve to partition S» the edge V; Vs and edge V;V> 
can be combined, thus avoid creating a new cut. 
Formally, the CDG algorithm is presented below. 


CDG algorithm: 
[1 ]Initialize Step: 
1: for all the vertex v in graph do assign(v) = 0 
[2|Greedy Step: 
1: for each v,in V do 
2: for each partition S; do 
3 (consider each w € N(v+) that has already been allocated) 
4 computing (v, S;); 
5 computing O(v, S); 
6: get score; by score function (6) 
7 
8 
9 
0: 





end for 
m = arg max { score; }, i € [1,k] 
add v, to partition S,, and set assign(v,) = m 


10: end for 


5.2 Affine Combining Deterministic Greedy Algorithm: 
A Online Heuristic with Affine Message Combining 


In this section we present an online heuristic algorithm Affine Combining Deter- 
ministic Greedy (ADG) algorithm for graph partitioning based on the cost function 
with affine message combining (Eq. 5). 

For a new arriving vertex, ADG computes and compares the scores (cost) on 
condition that assigning this vertex to every partition and select the partition with 
the optimal (minimum) score to assign the vertex. This procedure will be repeated 
until all the vertices have been assigned to the partitions. The score function used by 
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ADG is based on the cost function (Eq. 5). Besides, a balancing item (J — |S;lXn/k)) 
is incorporated to the score function to guarantee balancing. In addition, the item 
\N(v) NN(S;)| is also incorporated since it reflects the better partition from another 
angle. 

The heuristic score function used in ADG is 


Saag = [C (ax O(v, Si) + 1)] x 1 fu(S;) (7) 


Since the Scpg 1s already has the item y(S;) to balance the partition, the heuristic 
of ADG algorithm is simplified: 


Minimum: Saag 


Here cost C*"* is computed by Eq. 5. It denotes the message overhead incurred 
by adding a new vertex to the partition. Parameter a is a weight factor. 

Specifically, ADG includes the following steps: 

In the initial step, each of the vertices v (v € V) in Graph is set as not assigned 
(assign(v) = 0) and there is no message channel between any pair of the partitions 
(Cj, =0,i,7 €[1..k] and k is the number of partitions). 

In step of adding a new vertex v, we first get the information of v, including the 
neighbours of v, (N(v)) and edge information that indent to or from v, Then we 
evaluate each candidate partition and find the optimal one according to the score by 
assigning v, to the partition. 

In the process of evaluating each candidate partition, for example, evaluating the 
candidate partition 7, we deal with five different cases: 


(1) vertex w, the neighbour of v, is not assigned before; we just ignore w. 

(2) w is already assigned to the same partition i. In this case, if we assign v, to 
partition i, there will no message overhead incurred. 

(3) w is already assigned to a partition that is different with partition i. At the same 
time there is no message channel between those two partitions. We need to 
create a new message channel between those two partitions. 

(4) w is already assigned to a partition that is different with partition i. There 
already exists a message channel between those two partitions. If the edge 
v,w can be combined with a message in this message channel (according to the 
combining definition, they have the same source vertex or end vertex) we only 
need adding the length of this message. 


Otherwise (5), we need create a new message to utilize the message channel 
between partition i and p. 
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The ADG algorithm is described below: 


ADG algorithm: 
[1]Initialize Step: 

1: for all the vertex v in graph do assign(v) = 0 
[2]Greedy Step: 

1: for each v,in V do 

2: for each partition S; do 


3 (consider each w € N(v,) that has already been allocated) 
4 if w is also allocated in partition S$; 
S: computing O(v,S;) 
6 else 
7 computing CYS by equation 5 
8: compute the score; by score function (7) 
9: end for 
10: m = arg max { score; }, i € [1,k] 
11: add v, to partition S, and set assign(v) = m 
12: end for 


6 Experiment 


We present the experimental results of our algorithms in comparison with the 
Random, DG, LDG and Metis algorithms. It should be noted that Metis is the offline 
algorithm and the others are online algorithms. Here we compare the results with 
Metis since it is often used as an ideal reference to evaluate the performance of 
online partitioning algorithms. 

Below Sect. 6.1 describes the experimental setup. Sections 6.2 and 6.3 present 
our results using our online heuristic algorithms that employ two different cost 
functions, respectively. The efficiency of CDG and ADG is measured experimen- 
tally in Sect. 6.4. Finally, we further explore the performance of those algorithms in 
different stream orders. 


6.1 Experimental Setup 


The real-world graphs data used in our experiments are shown in Table 1. They are 
directed graphs and publicly available on the Web [20]. Some ineffective edges that 
include the vertex number out of the range of the definition on the dataset itself are 
removed, if any (Table 1). 


e Evaluation Criteria 


The performance of an algorithm is measured using two criteria from the 
resulting partitions: 


(1) The first criteria is widely used by state-of-the-art algorithms to measure the 
performance of graph partition: For a fixed partition X, we use the measures of 
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Table 1 Datasets used in our Graph Data Nodes Edges Description 

ape p2p-Gnutella04 | 10876 39994 | pop 
P2p-Gnutella05 8846 31839 P2P 
P2p-Gnutella06 8717 31525 P2P 
P2p-Gnutella08 6301 20777 P2P 
P2p-Gnutella09 8114 26013 P2P 


P2p-Gnutella24 26518 
P2p-Gnutella25 22687 
P2p-Gnutella30 36682 
P2p-Gnutella3 1 62586 
Slashdot08 11 77360 
Slashdot0902 82168 
Soc-Epinions 1 75879 
Soc-sign-Slash 71357 


65369 | P2P 

54705 | P2P 

88328 | P2P 
147892 | P2P 
905468 | Social 
948464 | Social 
508837 | Epinions.com 


516575 signed social 
network 


Wiki-Vote 7115 103689 | Wiki network 
Amazon0302 262111 1234877 | Co-purchasing 
Amzaon0601 403394 3387388 Co-purchasing 


the fraction of edges cut y and the normalized maximum load A. In addition, 
we use the measures of the fraction of combined edges cut Yeompine tO Measure 
the performance of each algorithm with message combing (That is to say, we 
take edges cut that can be combined as one edge cut). They are defined as 


Edge Cuts by X 
So ——— a 


(8) 
JE] 
Combined Edge Cuts by X 
Y combine = en | (9) 
Maximum Load 
A= ———_ 10 
> (10) 


Notice that n>y>n/k since the maximum load of a partition is at most n and 
there always exists at least one partition with at least n/k vertices. 


(2) The other criterion uses cost but the edge cuts to evaluate the relative 
improvement of competed algorithms over the random method. The cost of the 
resulting partition is computed by our new cost functions with combining. We 
define Cost ,, the cost of the partition by method m and take the resulting 
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partition of random as our base reference to evaluate other four methods. The 
relative cost ratio p is defined as 


B Cost, 
CoStrandom 


(11) 

Here p ranges from O (there is not any message cost in resulting partition) to 1 
(the resulting partition has the same message cost as that of random method). From 
this definition the smaller p indicates the less cost compared to that of random 
method and shows the better performance. 


6.2 Partition Results Using CDG Heuristic 


To evaluate the performance of our heuristic algorithm (CDG) based on cost 
function with message combining, we compare it with four state-of-the-art 
heuristics: Random, DG, LDG and Metis on condition using the same message 
combining cost function described in Sect. 3.1. 

To get the resulting partition with message combining of four competing algo- 
rithms, we first use those algorithms to get the resulting partition and then apply 
message combining to their resulting partitions. Specifically, to evaluate the effecting 
of message combing, we define y as the edge cut fraction (same with the definition of 
y) and y, as the combined edge cut fraction (same with the definition of Ycombine)- 

As shown in Tables 2 and 3 the performance (y,.) of CDG is close to Metis and 
outperforms Random and LDG algorithm. In case that the number of partitions 
exceeds 64 DG shows better results on y, than CDG but it performs very poor in 
load balance (A). The other four algorithms perform well on this issue. 

Table 4 presents detailed results including the edge cut fraction y, combining 
edge cut fraction y, and the load balance A on a dataset Slashdot0&11. CDG shows 
better performance than Random, LDG on all different numbers of partitions. It 
even shows better performance on combining edge cut fraction when the number of 
partitions is larger than 16. DG still shows the best results on both y and y, but the 
worst load balance. 


Table 2 Average combining edge cut fraction y, on different numbers of partitions 


K Average combining edge cut fraction y, 


Random 
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Table 3 Average load balance A on different numbers of partitions 


2 4 Js 6 82 |ø |i | 256 
Random 1.26 
DG 43.8 
LDG 1.01 

1.03 | 


K Average load balance A 
2 


CDG 1.00 1.01 
Metis | 1.08 1.02 


Table 4 Detailed results on Slashdot081 1 


K 2 Je |s |6 |32 tf 128 | 256 
Random |y |457 (68.6 |80 (85.8 |886 [90 [90.8 |911 
50.2 | 63.5 
A ho io fio fio fii fat fia [i2 
DG 4715 |34.2 
ye [2 |24 [39 6s [70o |77 [i04 [84 
A [16 |26 |44 (60 [128 [28.8 [|398 |17 
LDG 77.9 
39.7 | 50.2 


A [i 
CDG 84.1 |854 
v |i |29 [s9 [99 fis3 [2ni9 [292 [37 
A (10 (10 (10 (10 f10 f10 [10 [10 
Metis (y |226 [401 |515 [58.2 |629 |656 |68 | 70.2 
Ye [09 |25 (52 |96 [159 [|243 |34 |434 
; 0 l. 1.0 


6.3 Partition Results Using ADG Heuristic 


In this section, we present the partition results using ADG on the same dataset 
(Table 1) and compare it with Random, LDG and Metis. We did not compare with 
DG since it is an unbalanced method. We take random method as the reference and 
use relative cost ratio p to evaluate other algorithms utilizing the affine message 
combining cost function (Eq. 5). 

Specifically, to validate the performances on different settings of w;, w2 and w3 
described in definition (4), we set w; = 1 and use another symbol 7 = w;/w2 = w2/ 
w3 to adjust the relative ratio between w;, w2 and w3. 7 is greater than 1 since 
w; > W2 > w3 as we discussed in Sect. 4.1. Here we define 7 is ranged from 1 to 
128 with two-fold increase in each step. 

Experiments are performed on fixed number of partitions (k = 256). 

As shown in Table 5, ADG outperforms the online method LDG on all the 
setting of y. The gap between ADG and LDG is increasing as the 7. When ņ is larger 
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Table 5 Average relative cost ratio p of methods on 256 partitions using different combinations 
of wi, W2 and w3 


n Relative cost ratio p 


Random 
LDG 0.83 
ADG (071 |069  |0.64* _| 0.60% 
Metis 0.70 


“*”indicate the best average relative cost ratio 


a2 ja [z 
O Pe l 
0.89 | 0.90 
0.55* | 0.54% 
0.75 | 0.76 





than 16 (their gap is 29 % then), their gap is still increasing but the increasing speed 
is reduced. At the same time ADG even outperforms Metis when 7 is larger than 4 
and it reduces 20 % cost than Metis when y is 64. 


6.4 Runtime Efficiency 


All algorithms have been implemented in C++, and all experiments were per- 
formed on Linux computing environment. Each experiment is performed on a 
single core with Intel Xeon CPU at 2.2 GHz, and 64 GB of main memory. Wall 
clock time includes only the algorithm execution time, excluding the required time 
to load the graph into memory. 

According to Table 6, in terms of the time efficiency, CDG and ADG are 
comparable to other two online algorithms. Random method is the fastest method 
and Metis keeps near stable time consuming on different numbers of partitions. 


6.5 Performance Comparison in Different Stream Orders 


In a sense, a heuristic’s performance is also influenced by the stream order. Except 
random stream order which is generally used as default setting, the other two typical 
stream orders, DFS (Depth-First Search) and BFS (Breath First Search), are also 


Table 6 Runtime result of K Run time (second) 


CDG and ADG on 

Amazon0601 with different 2 |4 |s [16 |32 |64 |128 | 256 

numbers of partitions Random 1 

DG 12 17 
3 3 3 4 


we |a [s s [a [s 7 h2 fis 
os |a Js la |s |e 7u f9 
aoa |a |a |a |6 |s le [2 [20 
Meis |32 [33 |33 [33 [34 [34 |24 [os 
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often used to evaluate a heuristic [18]. To probe such influence on our proposed 
heuristics, we compared the performance of CDG and ADG and the competition 
heuristics on those three different stream orders: Random, DFS and BFS. 


Random: Vertices arrive according to a random permutation. 
DFS: We select a vertex uniformly at random. We perform depth-first search 
and sample the nodes with DFS. If there are multiple connected components, the 
component ordering is done at random. 

e BFS: This ordering is identical to the DFS ordering, except that we perform 
breath first search. 


(1) Comparison on average combining edge cut fraction (y,) and load balance (A) 


Table 7 shows the average combining edge cut fraction (y,) and load balance 
(A) of five competition methods on the data in Table 1 and on eight different 
numbers of partitions (from 2 to 256). 

As shown in Table 7, CDG outperforms the random and LDG methods on three 
stream orders. It has similar performance on random and DFS stream order and 
even better performance on BFS compared with Metis. DG method shows the 
worse performance on DFS and BFS stream orders. 


(2) Comparison on average relative cost ratio p 


To validate the performance of those heuristics in different stream orders 
according to the relative cost ratio p, we conduct the experiment on all the data 
described in Table 1. The number of partitions is fixed to 256. 

Table 8 presents the average relative cost ratio p of four competition meth- 
ods (DG method is not considered for it is an unbalanced heuristic) on fixed n (n = 
W1/W2 = W2/w3 = 64). ADG outperforms LDG and Metis on all the three stream 


Table 7 Average y, and A 
on different numbers of 
partitions 


Heuristic Random DFS 


Ye 
48.9 


w A 
Random 48.9 | | 
DG 265 | | 
LDG 32.9 | 
coc si 
Metis 25.6 | | 


Table 8 Average relative Heuristic = 
cost ratio p of methods on 256 aA a 
partitions using n = 64 andom C | oe 
ee 079 066 
= 


Metis 
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orders. At the same time the performance of LDG and ADG in DFS and BFS orders 
is significantly better than that in random order. It can be ascribed to that in DFS 
and BFS orders that the heuristics of LDG and ADG are easier to allocate the nodes 
and their neighbours to the same partition and thus reduce the consumption on 
message channel. 

Figure 4a—c presents the detailed results of the average relative cost ratio (p) on 
eight different n (1, 2, 4, 8, 16, 32, 64 and 128) in three stream orders. We can see 
that the average relative cost ratio of ADG is below than that of LDG in all stream 
orders. Moreover, ADG also shows better performance in all stream orders than 
Metis when n is larger than 4. Furthermore, the average relative cost ratio (p) of 
LDG and Metis is increased with n whereas that of ADG is stable. This further 
validates the advantage of adopting cost function with affine message combining in 
graph partition. 
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Fig. 4 Relative Cost Ratio (p) of Random, LDG and ADG in different stream orders. a Graph 
partitioning in random stream order. b Graph partitioning in DFS stream order. c€ Graph 
partitioning in BFS stream order 
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7 Discussion 


In this paper, we propose a cost function for graph partitioning that considers not 
only message combining but also the different types of messages. Based on this cost 
function, we develop two heuristics CDG and ADG for large graph partitioning 
problem. We evaluate our method extensively on real-world graphs over a wide 
range of number of partitions and verify consistently the superiority of our method 
compared to existing ones. Specifically, despite the fact that CDG and ADG per- 
form a single pass over the graph, it achieves performance comparable to and even 
better than Metis. Furthermore, our affine cost function can be dynamically adjusted 
to match the different realistic clusters to enhance the partition performance of the 
heuristics on different kinds of clusters. 

Currently, CDG and ADG are the first heuristics that utilized different kinds of 
message combining cost functions in graph partition. They have the potential to 
further improve the performance by more elaborately analysing the relation 
between graph and the different types of combining messages. 

Another interesting and potential future research problem is to derive an effective 
offline algorithm based on the affine cost function defined in this paper since it may 
be more fit for different messages consumed in modern clusters. 


Acknowledgments The work described in this paper was supported by the US NSF grant CNS 
1337899: MRI: Acquisition of Computing Equipment for Supporting Data-intensive Bioinfor- 
matics Research at the University of Kansas, 2013-2016. 


References 


1. Malewicz G, Austern MH, Bik AJ, Dehnert JC, Horn I, Leiser N et al (2010) Pregel: a system 
for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International 
conference on management of data, 2010, pp 135-146 

2. Avery C (2011) Giraph: Large-scale graph processing infrastruction on Hadoop. In: 
Proceedings of Hadoop Summit. Santa Clara, USA 

3. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing 
with working sets. In: Proceedings of the 2nd USENIX conference on Hot topics in cloud 
computing,pp. 10-10 

4. Low Y, Gonzalez J, Kyrola A, Bickson D, Guestrin C, Hellerstein JM (2010) Graphlab: a new 
framework for parallel machine learning. arXiv:1006.4990 

5. Shao B, Wang H, Li Y (2013) Trinity: a distributed graph engine on a memory cloud. In: 
Proceedings of the 2013 international conference on Management of data, 2013, pp 505-516 

6. Ke Q, Prabhakaran V, Xie Y, Yu Y, Wu J, Yang J (2011) Optimizing data partitioning for 
data-parallel computing. HotOS XII 

7. Kernighan BW, Lin S (1970) An efficient heuristic procedure for partitioning graphs. Bell Syst 
Tech J 49:291—-307 

8. Fiduccia CM, Mattheyses RM (1982) A linear-time heuristic for improving network partitions. 
In: 19th Conference on Design Automation, 1982, pp 175-181 

9. Feige U, Krauthgamer R (2002) A polylogarithmic approximation of the minimum bisection. 
SIAM J Comput 31:1090-1118 


114 X. Chen and J. Huan 


10. Ng AY (2002) On spectral clustering: analysis and an algorithm 

11. Karypis G, Kumar V (1998) A fast and high quality multilevel scheme for partitioning 
irregular graphs. SIAM J Sci Comput 20:359-392 

12. LUcking T, Monien B, Elsässer R (2001) New spectral bounds on k-partitioning of graphs. In: 
Proceedings of the thirteenth annual ACM symposium on Parallel algorithms and 
architectures, pp 255-262 

13. Abou-Rjeili A, Karypis G (2006) Multilevel algorithms for partitioning power-law graphs. In: 
20th International parallel and distributed processing symposium, 2006. IPDPS 2006, 10 pp 

14. Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. 
Commun ACM 51:107-113 

15. Kang U, Tsourakakis CE, Faloutsos C (2009) Pegasus: a peta-scale graph mining system 
implementation and observations. In: Ninth IEEE International conference on data mining, 
2009. ICDM’09, 2009, pp 229-238 

16. Gonzalez JE, Low Y, Gu H, Bickson D, Guestrin C (2012) Powergraph: Distributed 
graph-parallel computation on natural graphs. In: Proceedings of the 10th USENIX 
symposium on operating systems design and implementation (OSDI), 2012, pp 17-30 

17. Bourse F, Lelarge M, Vojnovic M (2014) Balanced graph edge partition 

18. Stanton I, Kliot G (2012) Streaming graph partitioning for large distributed graphs. In: 
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and 
data mining, 2012, pp 1222-1230 

19. Tsourakakis C, Gkantsidis C, Radunovic B, Vojnovic M (2014) Fennel: Streaming graph 
partitioning for massive scale graphs. In: Proceedings of the 7th ACM international conference 
on Web search and data mining, 2014, pp 333-342 

20. Leskovec J, Krevl A (2014) SNAP Datasets: Stanford Large Network Dataset Collection 


Big Data Analytics Platforms 
for Real-Time Applications in IoT 


Yogesh Simmhan and Srinath Perera 


Abstract Big data platforms have predominantly focused on the volume aspects of 
large-scale data management. The growing pervasiveness of Internet of Things 
(IoT) applications, along with their associated ability to collect data from physical 
and virtual sensors continuously, highlights the importance of managing the velocity 
dimension of big data too. In this chapter, we motivate the analytics requirements of 
IoT applications using several practical use cases, characterize the trade-offs between 
processing latency and data volume capacity of contemporary big data platforms, 
and discuss the critical role that Distributed Stream Processing and Complex Event 
Processing systems play in addressing the analytics needs of IoT applications. 


1 Introduction 


Internet of Things (oT) is an emerging architectural model that allows diverse 
sensors, controllers, devices, and appliances to be connected as part of the wider 
Internet [1]. IoT is driven by the growing prevalence of network-connected devices 
or “Things” that are present as part of physical infrastructure (e.g., Smart Meters 
that connect to Power Grids and Smart Appliances at residences [2]), which 
observe the natural environment (e.g., Air Quality Monitors in Urban Settings [3]), 
or monitor humans and society at large (e.g., FitBit for individual fitness tracking 
and Surveillance Cameras at event venues). In addition to monitoring and trans- 
mitting the observations over the network, these “Things” may also be controlled 
remotely, say to turn off a smart air conditioning unit or to turn on a smart vehicle, 
through signals over the network. As such, estimates on the number of such things 
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will be part of the IoT range in the billions, and these things may be generic, such as 
a smartphone, or specialized to a domain, as in the case of a smart cart in a grocery 
store [4]. Some definitions of IoT include Humans as first-class entities within the 
internet of things and humans (IoTH) [5]. 

IoT enables the generation of enormous volumes of data observed from the sensing 
devices, and this begets a key question of how to meaningfully make use of this 
information. The promise of IoT lies in being able to optimize the network-connected 
system or improve the quality of life of humans who use or interact with the system. 
However, this is predicated on being able to analyze, understand, and act on this 
information. This opens up fundamental questions on big data analytics that are still in 
the process of being characterized [6]. Such analytics may be simple correlations 
between the outside air temperature and the load on the power grid (1.e., hot days lead 
to increased use of air conditioners, and hence greater load on the power grid), to more 
complex causal relationships, such as an evening accident on a freeway leading to a 
progressive neighborhood which delays the jump in power usage caused by electric 
vehicles being recharged when commuters reach home. 

The ability to observe IoT systems does not mean that we can understand or 
reason about such complex systems, just as in the “butterfly effect,” the ability to 
observe butterflies does not allow us to link the flapping of their wings to a future 
tornado at a remote location [7]. However, the unprecedented ability to sense 
physical and natural systems does offer a unique opportunity to apply, extend, and 
invent data analytics techniques, algorithms, and platforms to help make decisions 
to improve and optimize such systems. In particular, it places emphasis on the 
Velocity dimension of big data [8] and consequently, on the temporal aspect of data 
generation and its analysis. This is intuitive. IoT devices are often active continu- 
ously, leading to a perpetual stream of time series data being emitted from them. 
Such temporal data are transient in value and IoT systems benefit if this data is 
analyzed and acted upon in real time to close the loop from network to knowledge. 

In the rest of the chapter, we introduce use cases and scenarios of emerging IoT 
domains that are grappling with big data and have a tangible societal impact. We use 
these to motivate specific analytics and techniques that are required by such domains 
to operate effectively. Subsequently, we discuss big data platforms that are making 
headway in offering such tools and technologies, and provide case studies of such 
analytics in action for IoT applications. We finally conclude with open problems we 
foresee for big data analytics in IoT, as this nascent field matures beyond its hype. 


2 IoT Domains Generating Big Data 


2.1 Smart Cities 


There is intense global interest in enhancing the quality and sustainability of urban 
life. This is driven by the growing urbanization in developing countries, particularly 
highly populated ones like China, India, and Brazil, which is stressing the urban 
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infrastructure and affecting the livability of residents, as well as the need to 
seamlessly integrate the urban ecosystem with a technology-driven lifestyle. The 
Smart Cities concept attempts to infuse sensing, control, and decision-making into 
different utilities offered in cities, such as smart transportation, smart power grids, 
and smart water management [3]. In this day and age, it also attempts to improve 
safety and health through urban surveillance and environmental monitoring for air 
and noise pollution. 

Smart power grids monitor the consumers of power using advanced metering 
infrastructure (AMI), also called Smart Meters, to provide real-time information 
about the amount of power consumed, typically at 15-min intervals [9]. This, 
combined with metering at neighborhood transformers and community sub-stations, 
allows the power utility to get a realistic view of their distribution network. The 
goal for the utility is to use this data to manage their supply and demand to avoid a 
mismatch that could cause brownouts and blackouts, as well as to switch their 
supply mix from reliable but polluting coal and gas-based power plants to 
less-reliable solar- and wind-based generation. Data-driven targeting of specific 
consumers helps in reducing consumption through demand-response optimization, 
rather than increase production, which is another goal. This requires intelligent 
shifting, shaving, and shedding of loads from household appliances, electric 
vehicles, and industrial units based on their load profiles. Smart water management 
likewise attempts this intelligent resource management for water, with the addi- 
tional goal of ensuring adequate water quality to mitigate health concerns. 

Smart transportation [10] uses sensors that monitor road traffic conditions, using 
inductive-loop traffic detectors on roads, traffic cameras, and in-vehicle and 
in-person monitoring devices, with the goal of using such data to efficiently 
managing traffic flow through smart traffic signaling. This also extends to managing 
public transport by optimizing the schedule of buses and trains on-demand, based 
on real-time data, and planning the routing of freight vehicles in transport hubs and 
port cities to ease the traffic. Such data collection, particularly through surveillance 
cameras, also helps with vehicular safety (tracking hit-and-run accidents, or auto- 
mobile theft), public safety (burglaries or attacks), and disaster management (tor- 
nadoes, earthquakes, urban flooding). Users already see the benefits of smart 
transportation through apps like Waze and Google Maps that offer directions based 
on real-time traffic conditions monitored through their smartphone apps, but getting 
a more integrated view over diverse sensor and human data for smart transport 
management is a particular challenge. 


2.2 Smart Agriculture 


With global warming affecting food supply and the increasing affordability of the 
world’s population to food affecting its demand, there is starting to be a stress on 
the agricultural output. While mechanized farming by industry-scale farmers is the 
norm in developed countries, a majority of farming in developing countries is still 
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human-intensive with low to negative profit margins. As such, technology and 
particularly IoT can play a role in improving practices at both these scales and 
maturity levels [11]. 

One of the key challenges in farming is to decide when to water the crops. Cash 
crops like vineyards are very sensitive to the soil moisture and humidity. The 
quality of the produce is affected by over or under irrigation, which also depends on 
weather conditions such as sunlight and warmth, rainfall, and dew conditions [12]. 
On the other hand, irrigation in developing countries like India relies on pumping 
groundwater, and the frequency of this depends on availability of intermittent 
electricity supply to operate pumps. As such, IoT can make it possible to use soil 
moisture data from ground sensors, remote sensing data from satellite imagery, and 
data from weather prediction models as well as supply schedule of power utilities to 
intelligently plan the irrigation of crops. 


2.3 Smart Health and Lifestyle 


Sports provide natural use cases for IoT, which, while less critical than other social 
scenarios, do provide early insight into emerging ideas due to rapid technology 
penetration. Common uses of IoT in sports rely on sensors placed in a player’s 
shoes, helmet, or clothing, which provide high-resolution data (e.g., x, y, z location, 
speed, acceleration) about the player’s actions coupled with vitals like heart rate. 
For example, the DEBS 2013 Conference’s Grand Challenge [13] is based on a data 
collected from players’ shoes and the ball in a soccer game. Also, American 
football teams have started placing cameras and sensors in players’ helmets to 
detect concussions, and one can even buy basketballs off-the-shelf with sensors 
embedded within to track plays. The potential benefits of having such fine-grained 
data on the players and the equipment can ensure player’s safety from injuries 
suffered at game time, better referee decisions, data-driven player selection (e.g., 
MoneyBall [14]), augmented TV broadcast with the enhanced game analysis, and 
even embedded virtual reality views for the audience. 

Health and lifestyle examples range from activity monitors and smart watches 
such as FitBit, Withings, and Apple Watch to in-home care for the elderly. There is a 
growth in electronic devices that track people’s behavior and basic health metrics, 
and enables them to get warnings about potential illness or a health condition, or 
just for personal analytics that help people reach fitness goals as part of a “quan- 
tified self’ movement [15]. The increasing prevalence of smart watches and smart 
health devices is leading to individual health data being collected, and generic or 
specialized apps that can monitor and analyze them. 

At the same time, such inexpensive sensors would make it possible to have 
sustainable in-home care for the elderly, recuperating patients, and those with 
long-term medical conditions to live at their home while having their 
health-monitored remotely [16]. Also IoT devices can improve the quality of 
hospital care and closer integration of case handling by monitoring medication to 
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ensure patients are given the dosages consistent with their prescription, and avoid 
nosocomial ailments by ensuring caregivers wash their hands after procedures. In 
developing countries that rely largely on tertiary care by community nurses, 
neo-natal monitoring through bracelets coupled with smartphones can help detect if 
a baby shows signs of trauma that needs immediate medical attention [17]. 


2.4 Smart Retail and Logistics 


Retail and logistics domains have a vital need to track their supply chain activity. 
For example, retailers are interested to track their inventory, shipping, and even the 
behavior of customers in their stores. RFID tags have been a major part of the 
supply chain for a while, with Walmart using them to handle their logistics [18]. It 
enables them to automatically track what items move in and out of the store without 
having to scan them, and to reliably know where each item is, avoiding operator 
errors. More generally, “Smart things” like RFID tags, GPS trackers, iBeacons, etc., 
can track items that are being transported and avoid costly errors in domains like 
airline baggage, postal vehicles, and even livestock and wildlife. 

In logistics, speed and accuracy are vital. Within a retail store, smart tracking can 
reveal a wealth of information about consumer behavior and provide a rich inter- 
active experience. Strategically placed sensors in store shelves and aisles can track 
what area of the store gets most attention and what regions confuse shoppers [19]. 
One often-overlooked sensor that is present in most stores is a video camera, 
typically used for security but which can, through streaming video analysis, also 
reveal customer behavior. On the other hand, mobile devices and Bluetooth 
low-energy (BLE) beacons can provide an interactive experience to the consumer, 
greeting them by name, helping them locate items, and even making suggestions on 
related products and discount coupons. These can provide mutually beneficial value 
to consumers and retailers, increase the sales volume for retailers, and help with 
efficient inventory management. 

Despite their huge potential, IoT devices and the communication technology that 
connect them are just enabling tools—only as effective as how they are used and the 
decisions that they empower. As a result, deriving actionable insights from data 
collected through IoT devices and carrying out appropriate actions though IoT 
devices or by other means is an open challenge for IoT-based systems. 


3 Role of Big Data Analytics in IoT 


The above scenarios present an overview of the potential application domains that 
would benefit from big data being generated through IoT infrastructure. Next, we 
explore in greater detail the role of analytics and decision systems in these use 
cases. 
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IoT systems are an example of autonomic systems that garnered significant 
attention in the early 2000 s [20]. Autonomic system design has resulted in several 
types of control loops, which often follow the MEAP model of Monitor, Analyze, 
Plan, and Execute operations. This is similar to the Observe, Orient, Decide and Act 
(OODA) control loop used in other domains. These are often closed loops, since the 
Execute or Act step would likely cause the environment to change, which would 
trigger the loop again, and so on. 

These feedback and control models are relevant within IoT applications as well. 
For example, a smart power grid application that performs home energy manage- 
ment may monitor the power consumption in the household and observe that the 
residents have just entered their home (e.g., the garage door opens), analyze their 
preferences, and decide to start the air conditioning unit that is configured to the 
optimum temperature level and even plan for coffee to be brewed in 15 min, by 
“talking” to smart appliances. Some of these scenarios are indeed possible, even at 
the present. 

However, there are many challenges in building intelligent systems that can make 
automated decisions and executing them. Some of the questions to consider are: 


1. How does it effectively learn from data, and dissociate signal from noise? 

2. How can it integrate expert knowledge with observed patterns? 

3. How can it understand the context (Where, When, Who, Where) and act 
accordingly? 

4. How can it comprehend the consequences of and interference between different 
actions? 

5. How does it plan for causality that are not instantaneous, but take place over 
time, across control iterations, and can fail? 


Of course, these are challenging problems not unique just for IoT but funda- 
mentally to many domains that can benefit from artificial intelligence, machine 
learning, and expert systems. However, the availability of big data in IoT domains 
makes them a particularly interesting candidate for exploration. 

There is a great deal of diversity in IoT use cases and their underlying decision 
systems, and these systems frequently contend with the above challenges. We can 
classify decision systems based on increasing levels of complexity. While this 
classification is not meant to be comprehensive, it illustrates the different degrees of 
control capabilities that are possible, and analytics that are required. 


1. Visual Analytics: Such techniques help humans analyze the data and present 
information to them in a meaningful form through a dashboard interface. They 
are designed to augment the human decision process with more information, 
presented in a cohesive and easily interpretable manner. An example is FitBit, 
which collects personal activity data using an in-person device and presents a 
summary of a user’s effort levels, estimated calories burnt, and progress toward 
monthly goals, through a mobile or web application. 

2. Alerts and Warnings: These engines allow decision logic to be provided by end 
users, and then uses these to interpret and classify data that arrives in order to 
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raise alerts or warning. These perform a certain degree of automated predefined 
analysis to help highlight situations of interest, which becomes critical when 
users have to deal with large volumes of data. For example, an environmental 
monitoring system may track the level of pollution or chemicals in an industrial 
city and send notifications on health hazards or chemical leaks to the citizens. 

3. Reactive Systems: Systems may also go one step further by taking concrete 
actions (beyond notifications) based on their decisions. Generally, they are 
designed with a rule-based language that describes actions to be carried out 
when certain conditions are met. For example, a smart lighting system may turn 
off the lights when nobody is present in a room. Here, there is a tight coupling 
between just two physical components—an infrared sensor and light bulbs, and 
along a single data dimension—the IR level that indicates human presence, L.e., 
the system is not aware of the bigger picture but just the local, unidimensional 
situation. 

4. Control and Optimize: Control systems operate in a closed loop where deci- 
sions lead to actions that are instantaneous, with the possibility that the actions 
can fail to meet the optimization goal. Control loop decision systems attempt to 
optimize the behavior of specific variables and also consider failure cases when 
deciding action outcomes. As discussed before, such MEAP or OODA systems 
are capable of generating a plan of action, executing the action, observing the 
response in a control loop, and recovering from a failure to meet a goal. For 
example, many electro-mechanical systems such as cruise control in a car or 
even a simple air conditioner thermostat operate on such a closed loop to either 
maintain the speed of the car or the ambient temperature. 

5. Complex Systems: Such systems understand the context and interaction 
between several decision loops and are capable of making high-level decisions 
that span multiple dimensions within a single domain. For example, systems that 
manage city traffic can discern interaction between multimodal transports such 
as Bus, Metro, and Trains, in scheduling the road and rail traffic signaling for 
efficient transit. 

6. Knowledge-driven Intelligent Systems: Complex infrastructure has cross- 
domain impact, with decisions in one domain impacting the other. 
Knowledge-based intelligent systems attempt to capture the relationship between 
different domains, such as transportation and power, or environmental conditions 
and healthcare, and optimize the decisions across these domains. The knowledge 
base itself is often specified by experts, and these may partially guide automated 
“deep learning” of correlations and causations within and across domains. 

7. Behavioral and Probabilistic Systems: Human being are an intrinsic part of 
IoT though they are often overlooked. As such, they are both sources of data and 
means for control, through messaging, suggestions, and incentives. Behavioral 
systems attempt to include human models as a part of the overall IoT system, 
with likely nonuniform behavior. As a generalization, probabilistic and fuzzy 
systems incorporate nondeterminism as an integral part of decision-making, 
which goes above and beyond failures (Fig. 1). 
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Fig. 1 Real-time decision-making 


These systems vary from simple analytics performed in a batch mode, to 
real-time detection of patterns over data to issue alerts, to make complex plans and 
decisions that may have downstream consequences over time. The temporal 
dimension and the context are two particular characteristics that make these systems 
challenging. In the next sections we discuss some of these concepts in detail. 
However, behavioral systems and beyond are still in their infancy. 


4 Real-Time Big Data Analytics Platforms 


Decision systems that go beyond visual analytics have an intrinsic need to analyze 
data and respond to situations. Depending on the sophistication, such systems may 
have to act rapidly on incoming information, grapple with heterogeneous knowl- 
edge bases, work across multiple domains, and often in a distributed manner. Big 
data platforms offer programming and software infrastructure to help perform 
analytics to support the performance and scalability needs of such decision support 
systems for IoT domains. 

There has been significant focus on big data analytics platforms on the volume 
dimension of big data. In such platforms, such as MapReduce, data is staged and 
aggregated over time, and analytics are performed in a batch mode on these large 
data corpus. These platforms weakly scale with the size of the input data, as more 
distributed compute resources are made available. However, as we have motivated 
before, IoT applications place an emphasis on online analytics, where data that 
arrives rapidly needs to be processed and analyzed with low latency to drive 
autonomic decision-making. 
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4.1 Classification of Platforms by Latency and Throughput 


In Fig. 2, we classify the existing big data platforms along two dimensions: the 
average throughput of data processed per unit time (along the Y axis) and the latency 
time to process a unit of data and emit a result (X axis). Volume-driven platforms 
such as MapReduce scale to terabytes of data, and on an average, are able to process 
100 MB/s of data per second (top right of figure). But this is in a batch processing 
mode, so the minimum time taken to generate a result (even for small data sizes, such 
as in the bottom right of figure) is on the order of minutes. This is forced by the need 
to store data on distributed disks, which introduces I/O and network overheads but 
also ensures persistence of data for repeated analysis. As a result, the time between 
data arriving and useful decision being made is in the order of minutes or more. 

Databases, both relational and NoSQL, also use disk storage but can be used 
with indexes to support interactive searches. This guarantees persistence while 
avoiding the brute-force table scans of MapReduce platforms. OLAP technologies 
(e.g., Pentaho) and projects like Apache Drill build a layer on top of databases for 
interactive processing. For large datasets, indexing needs to be done a priori, 
introducing some additional latencies. In such platforms, the scalability of data 
throughput is sacrificed (10 KB-1 MB/s) in return for faster query response times 
on the order of seconds to minutes. 

Stream Processing systems are also called “Real-Time processing” although they 
are in fact only low-latency (but not real-time) processing, in the strict sense of the 
word. Such systems can process data as they arrive at high rates by only keeping a 
small subset of the data in memory or on disk for the duration of processing, and 
can produce results in milliseconds. Here, the incoming data is transient and once 
processed is discarded (or separately stored in a NoSQL database for offline pro- 
cessing). Such platforms may operate on a single machine or on distributed sys- 
tems. These are the focus of further discussion in this chapter. 
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Fig. 2 Categorizing big data platforms based on latency of processing and size of data processed 
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Finally, In-Memory Computing uses the idea of loading all the data into dis- 
tributed memory and processing them while avoiding random access to distributed 
disks during the computation. Platforms like Apache Spark offer a streaming batch 
model that groups events within a window for batch processing. This, while sac- 
rificing some latency, helps achieve a higher throughput across distributed memory 
and can be 100’s to 1000’s of times faster than stream processing. 

Among these technologies, streaming analytics plays a key role as a layer that 
can perform initial, low-latency processing of data available from IoT devices. 
Their results may directly lead to actions, stored for future processing, or feed into 
models that take decisions. 

We next introduce specific technologies to support real-time analytics, and 
consider two use case scenarios as running examples. (1) Sports Analytics for 
Soccer Games, where sensors placed in the football players’ boots and the goal- 
keeper’s gloves generate event streams at 60 Hz frequency that contain time, 
location, and speed information. This is motivated by the DEBS 2013 Challenge 
[13]. (2) Analytics for Smart Energy Management in buildings that identify 
real-time opportunities for energy reduction using events from smart power meters 
and electrical equipment. This is motivated by the Los Angeles Smart Grid Project 
[9]. In the subsequent section, we discuss how these big data stream processing 
technologies are integrated into these use cases. 


4.2 Online Analytics to Detect Exact Temporal Patterns 


In many domains, the knowledge from domain experts and from past experiences 
provide us with interesting patterns that can occur, and such patterns (or signatures) 
may indicate situations of interest that require a response as part of a decision support 
system. We can detect the occurrence of such patterns and conditions using Complex 
Event Processing (CEP). CEP allows the specification of queries over one or more 
event streams, and consequently helps define such patterns in a formal manner. The 
queries are described through an Event Query Language (EQL) that is specific to the 
CEP engine, such as WSO2’s CEP Siddhi language or Esper’s EQL. The survey of 
Cugola and Margara [21] offers a detailed review of these query models and engines. 
Following are examples of some such patterns of interest: 


e Filter ambient temperature measurement events from an air conditioner 
(AC) unit if it crosses a threshold temperature. 
Maintain a moving average of the energy consumption over | h time windows. 
Correlate events from multiple streams by performing a Join operation across 
them based on an attribute, such as the room location of a power meter gen- 
erating consumption events and an AC unit generating temperature events. 

e Detect a temporal sequence of event patterns specified as a state machine over 
events, such as the occurrence of several power consumption events of a low 
value followed by a spike in their value indicating a heavier energy load. 
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e Preprocess events, such as transforming their units or dropping unused attri- 
butes of the events. 

e Track the change in state over space and time of an entity, such as the location 
of a football. 

e Detect trends captured by sequences, missing events, thresholds, outliers, and 
even complex ones like “triple bottom” used in algorithmic financial trading. 


As a generalization of these examples, CEP systems support the following 
categories of guery operators, shown with a sample CEP query definition using the 
Siddhi EQL. 


1. Filters and Transformations—This filters events from an input streams based 
on a predicate and places them, possibly after some transformation, into an 
output stream. 


e from PowerStream[consumptionKWH>10] insert into 


HighPowerStream 
e from ACUnitStream#transform.Op( (temperatureF-32) 
*5/9) 


insert into ACUnitMetricStream 


2. Window and Aggregation operations—This operator collects data over a 
window (e.g., period of time or some number of events) and runs an aggregate 
function over the window. 


e from PowerStream#window (60 min) select avg 
(consumpt ionKWH) 


3. Join two event streams on an attribute—This operator matches events from 
multiple data streams or data from the same stream based on a shared attribute 
value, e.g., location. This is similar to a database join, but performed on a 
window of events, e.g., of 1 min duration. 


e from PowerStream#window (1 min) join ACUnitStream 
on PowerStream.location = ACUnitStream.location 


4. Temporal sequence of event patterns—Given a sequence of events, this 
temporal operator can detect a subsequence of events arranged in time where 
some property or condition is satisfied between the events, e.g., the consumption 
KWH between the events is greater than 10. This can also use regular expression 
matches between events, say, to detect a pattern of characters from a given 
sequence of characters. 


e from every e = PowerStream 
-> PowerStream[e.consumptionKWH - consumptionKWH > 10] 
insert into PowerConsumptionSpikeStream 


Just as SQL provides a high-level abstraction over database queries, advanced 
CEP queries can be composed by combining the above four basic query types to 
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detect complex conditions. They also support the inclusion of user-defined functions 
as part of their transformations to allow more sophisticated and domain-specific 
operators like machine learning or forecasting algorithms to be embedded within the 
CEP processing. For example, the bounds generated by a regression tree learning 
algorithm can be used as a filter condition within a CEP query. 

A CEP Engine is a middleware server that executes CEP queries over event 
streams and produces a resulting stream of output events. The engine constructs an 
execution graph based on a given user query, where nodes in the graph represent 
the basic operators (e.g., Filter, Window) and edges in the graph represent event 
streams. The output event stream of a basic query can form the input stream to 
another basic query, thus allowing complex CEP queries to be composed. When a 
new event arrives at a stream, the event is submitted to all operators attached to that 
stream. Each operator processes the event and places the result on its output 
streams, which will then be delivered to downstream operators to which those 
output streams are connected to. 

CEP engines offer fast execution of CEP queries due to their in-memory model 
of execution that does not rely on disk storage. Each of basic operators is specif- 
ically optimized using specialized algorithms. Also, CEP engines rely on compact 
memory efficient and fast in-memory data structures. The fast execution allows for 
low-latency (“real-time”) processing of input event stream so that the engine can 
keep up query execution even with input stream rates. The small memory footprint 
also ensures that window queries can span a wide time or event window. CEP 
engine also provides various other optimization, as discussed in literature [22], such 
as 


1. Fast event flows between distributed operators, 

2. Multi-threaded executions with minimal concurrency control required from the 
user, 

Enforcement of event ordering even with data-parallel parallel execution, 
Throttling and rate limiting, and 

5. Checkpoints and recovery for resilience. 


— 


As an exemplar of how a CEP engine works, we briefly consider the Siddhi CEP 
engine. Siddhi queries are defined in the format illustrated earlier. The engine 
analyzes those queries to build a processing graph as discussed above. Input events 
are placed on incoming edges in the graph, the operator processes them and sends 
them to the next operator(s) if the condition matches, and eventually the final events 
arrive at the output edges of the graph. Operators are stateful. For example, a 
window operator will remember events it has received until conditions for the 
window trigger has been fulfilled. Most of the complexity of the CEP engine is in 
its operators, and they are crafted to be both CPU and memory efficient. 
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4.3 Distributed Stream Processing 


While CEP allows analytics of interest to be captured and detected in real time from 
streams of events, the process of ensuring the event streams is made available in an 
appropriate form falls within the space of stream processing. Stream processing 
allows composition of real-time applications as a directed acyclic graph (DAG), 
where vertices are application logic tasks, while edges are streams of events. 

Two distinctions between stream and complex event processing are that the 
events in CEP are strongly typed, and the queries conform to well-defined logic that 
operate on typed events and are visible to the CEP engine to perform optimizations. 
In stream processing, on the other hand, the event contents as well as the appli- 
cation logic are opaque to the stream processing engine, and the engine is primarily 
responsible for moving event payload between application logic and the execution 
of the logic over the events. The events are just a binary or text payload that the 
engine does not examine. In that sense, one can consider CEP as specialization of 
stream processing systems. 

Distributed Stream Processing (DSP) has gained interest recently due to the 
increasing quanta of real-time data that arrives and the hardware limits of 
single-machine, shared-memory execution. In DSP, the tasks of the DAG can be 
distributed across different machines in a local area network (LAN) or a Cloud data 
center, with event streams passing between them. Apache Storm is an example of 
such a system. 

DSP are faster than CEP and offer a lower latency for processing since they 
avoid the overheads of introspecting the contents of the message. Consequently, 
they are used as preprocessors for cleaning and ingesting data from sensors, and 
their outputs can be passed onto online analytics platforms such as CEP, or stored 
for offline analytics. Such preprocessing often takes the form of an extract—trans- 
form—load (ETL) pipeline used in data warehousing or extract—transform—analyze, 
except that it is done in near real time. These pipelines can help convert the format 
and representation of the data, detect and clean outliers, perform sanity checks and 
quality control, etc. on the incoming data. DSP also allow analytic logic to be 
embedded within them as a task, so that a CEP engine or a machine learning logic 
can be a task within a DSP application. 

For example, in the figure below, data streaming from hundreds of building 
sensors in a binary format is parsed to extract relevant information, and then 
transformed into a standard format necessary for analysis, a copy stored in a 
NoSQL archive for future processing, while a duplicate copy is forked for real-time 
analytics by a CEP engine (Fig. 3). 

Despite the events and task logic of the DAG themselves being opaque, DSP 
engines support several different semantics when composing streaming applica- 
tions. These include whether to perform a duplicate or interleave/split of outgoing 
events from a task, and using transparent keys with opaque values to perform a map 
operation from one task to a set of downstream tasks. Tasks may also perform either 
a merge or a join of multiple incoming messages, where the former interleaves 
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incoming messages while the latter aligns messages based on arrival order. Tasks 
may be stateful or stateless, which determines whether the state from processing a 
message is available to upstream events that arrive. Often, DSPs do not guarantee 
ordering of events as distributed coordination is more difficult and favors faster 
execution instead. 

The selectivity of a task determines the number of events it generates when 
consuming a single event, and its latency is an estimate of time taken to process one 
event. DSPs have a natural ability to pipeline execution of events. The length and 
the width of the DAG, respectively, determine the total latency of the critical path 
of execution, and the number of parallel tasks at the widest width of the DAG which 
can be exploited for task parallelism, if preceded by a duplicate task semantic. 
Stateless tasks can also leverage data parallelism by adding more processors to 
execute a single task, either on multiple threads in a machine, or across different 
machines in a cluster or virtual machines in the Cloud. This is more challenging for 
stateful tasks. 

More recently, there is interest in distributing such stream processing engines to 
go beyond a single data center into distributed execution across edge devices and 
the Cloud. This Edge + Cloud model of execution is motivated by the growing 
prevalence of generating data from distributed IoT sensor devices that themselves 
have computing capabilities [23], albeit limited. Devices like smartphones, Rasp- 
berry Pi, and Arduino running mobile processors and hosting sensors can perform 
part of the data filtering on-board rather than push all data to the Cloud or to a 
centralized data center for processing. This has several benefits. It limits the need to 
transfer large quantities of data that can incur communication costs. It also reduces 
the network round-trip latency that is paid when sending the data to the Cloud for 
analysis and receiving a response to take action. Finally, it enhances the privacy of 
the data sources as it allows for local analysis on-site rather than pushing it to a 
public Cloud. At the same time, there may be some analytics such as aggregation 
and comparisons across sensors/devices that require limited event streams to be 
centrally accumulated for analysis. This distribution of the computation across edge 
and the Cloud to meet the different metrics of costs, time, and privacy is of growing 
research interest as IoT applications start to actively utilize DSPs. 
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5 Real-Time Analytics Case Studies 


We next expand on the earlier example IoT domains of Smart Energy Management 
and Sports Analytics, and discuss how CEP and DSP systems are used for real-time 
analytics within these applications. 


5.1 Analytics for Smart Energy Management 


Smart power grids use information gathered from sensors that monitor the elec- 
tricity transmission and distribution infrastructure to ensure reliable, efficient, and 
sustainable generation and supply of power. In addition to individual residences in a 
city-scale electric grid, there are also islands of micro-grids that are operated by 
large institutional campuses such as universities, office parks, and heavy industry. 
Such micro-grids control the power distribution within their campus, with local 
cogeneration as well as electricity supply from the city grid, and offer the ability to 
deploy IoT solutions to instrument, monitor, and control the micro-grid behavior in 
an intelligent manner. 

The University of Southern California’s (USC) Campus Micro-grid, part of the 
Los Angeles Smart Grid project, is an example of such a model where stream and 
CEP platforms are used [9] with similar efforts taking place at other institutional 
campuses worldwide. Diverse sensors to monitor the micro-grid range from smart 
power meters that measure KWh, power load, and quality, to building area net- 
works that can monitor and control HVAC (heating, ventilation, air conditioning) 
units, lighting, and elevators. In addition, organization, spatial and schedule 
information on the departments, people, buildings, and class schedules are also 
available. 

These provide a mix of slow (buildings, departments) and fast (sensor readings, 
schedules) changing data that has to be analyzed in real time for smart power 
management. Specifically, the analytics aims to perform demand-response opti- 
mization, wherein the power consumption of the micro-grid has to be curtailed on- 
demand when a mismatch between the available power supply and expected power 
demand is identified. Stream and complex event processing platforms play vital a 
role in this decision-making process. 


(1) Information integration pipeline: Data coming from thousands of diverse 
sensors have to be preprocessed before analysis can be performed on them. 
Preprocessing includes extracting relevant fields from the sensor event stream, 
applying quality correction, performing unit transformations, and annotating 
them with static information available about that sensor. These steps are 
performed by an information processing pipeline that is composed using a 
distributed stream processing platform [24]. The outcome of the preprocessing 
is further forked into three paths: one, for semantic annotation and archival in 
a RDF data store for offline querying, analysis, and visualization; two, for 
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performing demand forecasting over KWh events that arrive from smart power 
meters; and three, for detecting power demand and curtailment situations using 
a CEP engine to aid decision-making. 


(2) Demand forecasting using time series: One of the paths taken from the 


information processing pipeline is to predict future energy demand by each 
building in the micro-grid based on past behavior. For this, we use the ARIMA 
time series forecasting model that uses the recent history of KWh events from 
a building for its prediction. The ARIMA model is included as a logic block 
within the distributed stream processing pipeline and the forecasts are gen- 
erated as a series of events. These are further analyzed in real time as part of 
the DSP to see if the sum of impending demand from different buildings is 
greater than the available generation capacity during that future period. This 
helps in deciding if a power curtailment action is required within the 
micro-grid. 


(3) Detecting power demand and curtailment opportunities: Another event 


stream that forks from the preprocessing pipeline is to a CEP engine, which 
appears as a logic block within the DSP. This CEP engine has two types of 
analytics queries registered with it, one to detect energy spikes that may occur 
due to special circumstances that cannot be captured by the time series model, 
and another to detect potential energy leaks that can be plugged to reduce 
energy consumption when required. We offer a few examples of these two 
classes of CEP queries [25]. 


Find all labs where a class is scheduled to take place in the next 15 min and the 
current power consumption by that lab is < 100 Wh. This indicates that the lab 
rooms can expect to draw more power once the class starts due to teaching 
equipment being turned on and a higher room occupancy. This queries the 
classSchedule event stream and the smartMeter event stream and can be 
implemented as a temporal sequence. 


from c = classSchedule 

->Time[c.time - now < 15m] 

->smartMeter[c.bldNo == bldNo and power < 100kW] 
insert into spikeStream 


Find all buildings where the ambient temperature is below the setpoint tem- 
perature for the HVAC, and the ambient temperature has increased by more 
than 5’F over the past 1 h. This indicates that the buildings where the HVAC is 
in low power mode may switch to a high power mode if the ambient temperature 
continues to rise. This queries the roomTemperature and the HVACSetpoint 
event streams and can be implemented as a temporal sequence. 


from t = roomTemperature 
->roomTemperature[at - at > 5] 
->HVACSetpoint[t.at < setpointT] 
insert into spikeStream 
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e Find all labs where a scheduled class has completed 15 min ago, no classes are 
currently scheduled and the current power usage is > 100 Wh. This indicates a 
potential for energy savings due to equipment that has not been turned off after a 
class has completed. This queries the classSchedule and smartMeter event 
streams. Here the NOT operator (!) checks if such an event has not happened. 


from c = classSchedule 
->t = Time[now - c.time > 15m] 
->!classSchedule[startT <t.now and endT > t.now] 
->smartMeter[c.bldNo == bldNo and power > 100kW] 
insert into curtailmentStream 


e Find all buildings where more than 6 HVAC units are active at the same time 
for more than 10 min each. This queries the HVACStatus event stream to 
identify buildings where HVAC units can be duty-cycled to operate in a 
round-robin fashion, thereby flattening the energy load across time. This is 
handled with two queries. 


from u = HVACStatus [action=on] 
->!HVACStatus [unitid == u.unitid and action=off] 
->t = Time[time - u.time > 10m ] 

select u.bid, u.uid, t.now 

insert into HVACUnitOn10m 


from HVACUnitOn10m 

select bid, count(uid) as count group by bid 
having count > 6 

insert into curtailmentStream 


5.2 Sports Analytics 


The Distributed Event-Based Systems (DEBS) Conference 2013’s Event Processing 
Grand Challenge offers a novel IoT use case for soccer analytics that is addressed 
using CEP platforms. Data for this use case came from a Soccer game that was played 
with sensors placed in the ball, the players’ boots, and the goal keeper’s gloves. Each 
sensor generates events which describes the current location (x, y, z), the timestamp, 
velocity (vx, vy, vz), and acceleration (ax, ay, az), at the rate of 60 Hz. 

This data can provide complete information of the game play. However, 
understanding and deriving higher levels events such as kicks, passes, ball pos- 
session, and offside requires detecting complicated temporal queries from these raw 
events. As part of the challenge, we implemented those queries using the Siddhi 
Complex Event Processing engine, and achieved throughput in excess of 140,000 
events/sec [26]. We look at some example queries to understand how the afore- 
mentioned CEP operators are combined to detect such complex sport strategies. 

Let us take Ball Possession as an example. In a soccer game, it is not fair to 
measure players just by the number of goals they have scored as defensive players 


132 Y. Simmhan and S. Perera 


get fewer chances to score. Ball possession, the time duration for which a player 
controlled the ball, is a much better predictor of a player’s performance. 

To detect ball possession, we first need to detect kicks on the ball. To do that, we 
used the hypothesis that if the player and the ball are within one meter of each other, 
and the ball’s acceleration has increased by more than 55 m/s~, then player has 
kicked the ball. This condition can be detected by combining the player position 
event stream and the ball position event stream using a join query as follows. Note 
that getDistance() is a user-defined function that uses a pair of (x,y,z) locations to 
determine the Cartesian distance between the locations, in meters. 


from Ball#window.length(1) as b join 
Players#window.length(1) as p unidirectional 
on debs:getDistance(b,p)< 1000 and b.a > 55 

select ... 

insert into kickStream 


Once we have detected kick events that have occurred on the ball by specific 
players, we can detect the ball possession by each player. Player A has ball pos- 
session for the duration between the first time that s/he kicked the ball, followed by 
any number of kick events by the same Player A, and terminated by a kick event by 
some other player. This can be detected using a temporal event sequence on the 
kickStream as given below. This query is a temporal sequence where commas after 
‘from’ separate an event or a condition about an event in the sequence. 


from old = kickStream, 
b kickStream[old.pid != pid ], 
n kickStream[b.pid == pid]*, 
el = kickStream[b.pid != pid ] or 
e2 = ballLeavingKickStream 

select ... 

insert into ballPossessionStream 


Such analysis can variously be used for sports management studies on the 
effectiveness of game play and the players, to build a dashboard on game status, to 
generate alarms about critical events in the game (e.g., missed offside), and 
real-time game analysis and predictions about the next move. 


6 Discussion and Open Problems 


The use cases we have presented exemplify the rich and novel information space 
that IoT applications offer for analytics and decision-making. These have a tangible 
impact on both technology and human lifestyle. Real-time big data platforms such 
as distributed stream processing and complex event processing that we have dis- 
cussed are two of many platforms that can help coordinate, process, and analyze 
such information to offer actionable intelligence that offer the “smarts” to these IoT 
applications. However, as we have observed, these platforms need to work in 
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tandem with other advances in computing, in areas like machine learning, data 
mining, knowledge harvesting, deep learning, behavioral modeling, etc. to provide 
holistic solutions that are robust and sustainable. 

There are several open challenges on fast data processing platforms themselves. 
Incorporating semantics into processing events is important to bring in contextual 
information about diverse domains into the queries and combine offline knowledge 
bases with online data that stream in. Such contextual information include schedule 
and environmental information, habits, and interests that are learned, and proximity 
with other entities in the virtual and physical space that help integrate with an active 
world of humans, their agents, and things. 

While the scalability of high-velocity big data platforms on captive commodity 
clusters is well studied, making use of elastic Cloud computing resources to 
on-demand adapt the execution to runtime changes in the application logic or 
incoming data rates is still being investigated. The nominal utility cost of stream 
and event processing and the real cost paid for Cloud computing resources will also 
come into play. 

Smart devices that we carry on our person or are deployed on physical infras- 
tructure have capable computing power despite their low power footprint. 
Enabling DSP and CEP engines to effectively leverage their compute capability, in 
conjunction with more capable computing resources at centralized data centers and 
Clouds, is important. This is sometimes referred to as fog computing or mobile 
Clouds. This has the advantage of reducing the round-trip latency for processing 
events that are often generated by the sensors, and actions taken based on the events 
have to be communicated back to actuators that are collocated with the sensors. 
This can also be robust to remote failures by localizing the closed-loop 
decision-making. Further, issues of data privacy can also be handled in a person- 
alized manner by offering the ability to select the computing resource to perform the 
analytics and decision-making on. 

Decisions that are driven by these analytics can have an immediate effect or may 
impact future actions. Defining performance requirements for such fast data pro- 
cessing platforms that take into account the lead time for decision-making and the 
resulting actions themselves can help prioritize processing of information streams. 
Having policy languages that can capture such dependencies, and real-time pro- 
cessing engines that can interpret and adapt to such needs, will help. 

There are several open challenges specific to CEP engines too. First is scaling 
CEP execution beyond single machine. Several systems use data partitions, 
either derived automatically or given explicitly by the users, to break the execution 
into multiple nodes [22, 27]. For IoT use cases, handling out of order events is a 
major challenge. With thousands of sensors without a globally synchronized clock, 
events can arrive from sensors out of sync, and many CEP operators that are 
sensitive to the event ordering will find it challenging to enforce order and also 
ensure runtime scalability. Google’s MillWheel is an effort to address this problem 
[28]. Finally, it is unavoidable that some of the sensors or readings may be faulty in 
an IoT deployment. Taking decisions despite those uncertainties is another chal- 
lenge [29, 30]. 
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Complex Event Processing in Big 
Data Systems 


Dinkar Sitaram and K.V. Subramaniam 


1 Introduction 


Complex event processing can be applied to solve problems arising in many 
interdisciplinary areas of computing where the data is obtained from different 
sources and at differing granularities both in time and space. This is different from a 
traditional store and process architecture where the data is first stored in a database 
and then analyzed. Given the extremely high data rate that some of the Big Data 
applications produce, the event processing model has provided an alternative to the 
store process model. Twitter, for example, has replaced the store process model by a 
distributed content-based publish—subscribe model [16]. Similarly, Google photon 
[5] mines event streams from user clicks and query terms to determine if an 
advertisement was associated with a search term; this is used to report ad clicks to 
the advertiser and helps Google monetize advertisement revenue. 

The above examples are both from social media and search, and are examples of 
real-time stream processing with simple event models. In this chapter, we will look 
at an alternative big data application—manufacturing intelligence—as a motiva- 
tional example to help illustrate the need for complex event processing. The tidal 
race data stream warehousing solution [22] considers the semiconductor manu- 
facturing example and highlights the need for complex analytics processing on a 
real-time system. They argue on the need for weaker consistency in processing the 
real-time processing streams to make progress over time. In this chapter, we argue 
that there is great potential in integrating complex event processing models with 
stream processing to gain meaningful insights on the manufacturing process. 
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2 Manufacturing Intelligence 


Consider the wafer cleaning process being utilized in the semiconductor industry 
[33]. This consists of passing the semiconductor wafer through a series of tanks as 
shown in Fig. 1. At each stage, the material is subject to cleaning at particular 
temperature and pressure with a chemical mixture. To ensure that the concentration 
of the chemicals is kept constant, the chemicals are replenished at periodic intervals 
possibly every 4—5 runs. Also, the temperature and pressure during a particular run 
are not necessarily kept constant and are varied based on the need of the process 
being employed. For example, it is possible that the temperature may be slowly 
raised to reach the required value over a period of a minute at a particular rate as 
illustrated in Fig. 2, but the pressure of liquid may be sprayed on the material in 
bursts at different rates across the time period. For example, the machine may be 
configured to spray two bursts in the first 10 s and then again fire three more bursts 
after a small gap as illustrated in Fig. 3. 

The intensity and the timing of the bursts are both critical in ensuring that 
cleansing process is able to remove the impurities from the substrate. Further, given 
a large manufacturing system, there will be multiple different machines and each 
machine may have different parameters that the process control engineer would 
vary in order to obtain optimal yield. Any deviation from these could potentially 
lead to poor yield. 





Fig. 1 Schematic of multi-tank cleaning system employed in the semiconductor industry 
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Fig. 2 Variation of temperature over time during a cleaning cycle (for illustration) 
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Fig. 3 Variation of pressure over time during a cleaning cycle (for illustration) 


A manufacturing intelligence system must be able to receive data from various 
types of sensors (pressure, temperature and, liquid flow rate), identifier tags such as 
RFID sensors and bar code sensors that identify the substrate material and be able to 
notify the process engineers of any deviations from the process. It must additionally 
also be able to keep track of all materials that underwent the deviated process. Note 
that not only the variety of data is large, but also the granularity of relevant 
information is varied across time and space. For example, temperature/pressure 
sensors are valid only for one cleaning session. However, replenishment of the 
chemical mixture must happen across multiple cleaning cycles. 

Traditional DBMS-based solutions will not be able to handle the volume and 
velocity of the data due to additional store/retrieve operation that is required to be 
performed in the critical path. For example, the temperature pressure sensors are 
typically required to be read every 50 ms, and a write operation across a network 
could potentially take longer than 50 ms thus building up a backlog of write 
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requests that are queued. Further, given a large factory setting, with a multitude of 
such machines, the velocity of data ingestion requires non-traditional means. 

While this appears to be an example of an IoT (Internet of Things) system, a 
more careful examination of the system reveals that the intelligence is derived from 
not only sensors that read data but also a mixture of cameras that take images for 
tracking quality, process configuration files, and log files produced by the con- 
trolling machines. In other words, given the volume and variety of the data that is to 
be ingested, it is better to treat the system as a big data system so that the analysis 
can derive value for the manufacturing unit. 

The above considerations necessitate the introduction of systems that can pro- 
cess and draw inferences from the events in real time. As argued earlier, for many 
applications, an event model that captures the relationship between the different 
events is needed. In the next section, we give a brief overview of various features of 
such a complex event processing system. 


3 Basic Features of Complex Event Systems 


A real-time complex event processing system that is capable of processing millions 
of events from various types of sources that can be viewed as illustrated in Fig. 4 
[12]. The event observers or the sources on the left generate events. In our 
multi-tank cleaner example, the sources refer to ttmestamped values of temperature, 
pressure, and batch codes of the wafers being processed. The brokers or event 
processing agents encode the business logic to act upon the events. For example, to 
decide if the temperature reached its critical value over an interval, the brokers have 
to process a sequence of values to determine if the operating temperature is valid. If 
so, it signals an event indicating the operating temperature which is valid. If not, an 
operating temperature invalid event is raised; this generated event can act as a 
source event for subsequent stages in the event processing network. Observe that 
the graph of the event processing network is logical and does not imply the physical 
distribution of the brokers. The processing agents can be distributed across a set of 
machines and need not be on a single machine. Also, this network must not have 
any cycles. There may be consumers that are interested in the events being gen- 
erated by the system. These are represented by the sinks in Fig. 3. 

A complex event processing system such as the one described above will have to 
consider the following aspects in designing the system. These are based on the eight 
requirements enumerated by [34]. 


1. Data Model/Semantics—this provides a mechanism to handle data from a 
variety of sources and also to handle relationships across various events. Con- 
sider our example of the manufacturing intelligence application. Herein, we 
need not only a model for being able to extract events from various types of 
sensors and log files, but also a model that can help express the relationship 
among the various events. 
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Event Observers Event processing 


(sources) agents (brokers) 
Event consumers 


(sinks) 





Fig. 4 Event processing framework 


2. Expression of Intent—the mechanisms through which the user specifies the 
rules or the relationships between the source events and the sink events. 

3. Performance/Scalability—since there are a large number of events to be 
consumed, the performance of event processing is critical to ensure that events 
are not dropped; hence, low latency of operations is critical. 

4. Fault tolerance—relates to the aspect that there may be missing data in the 
input due to network delays and also to the fact that some of the nodes pro- 
cessing the events may go down. The framework must be able to handle all such 
cases. 


We examine each of these requirements in detail in the following sections. 


4 Data Model/Semantics 


The data model provides a mechanism for the system to model various types of 
events that are being processed by the system. This can be separated into two parts 
—an input data model and a data model to express the relationship between various 
events. The former is described here in this subsection; the topic of the latter is 
treated in greater detail in separate section. 


4.1 Input Data Model 


Typically, the data model of the input event is kept fairly simple and the events are 
expressed mainly as a multi-valued tuple of the type 


<ts, source — id, key, value > 


where 

ts refers to the timestamp when the event was generated 
source-id_ refers to the source node that generated the event 
Key refers to the key associated with event 
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Value refers to the value associated with the key and the source at the time 
instant ts 

For example, in our multi-tank cleaner example, the source event from the 
temperature sensor could look like < 2388301, cleanerl, temperature, 
45.2 > which indicates that the value read from the temperature sensor is 45.2 
degrees and is associated with cleanerl1. 

This representation permits extension of the model to scenarios where the value 
need not just be a single reading. For example, the value could even be an image 
file or a URL. This model also ensures that the system is able to take into account 
scenarios where only a single value is associated with a point in time and hence 
eliminates redundancy from the network. 


5 Expression of Intent 


The event processing nodes or brokers will have to ingest the incoming events and 
process them according to the business rules setup by the user. Various approaches 
have been taken to express the user intent to process the incoming events such as 
using a query language, providing a programming paradigm or ability to run data 
analysis algorithms on the events. 


1. Stream Query Languages and Algebra—In this approach, a query language is 
defined that operates on a stream of events. The Telegraph CQ project [10] 
defines a continuous query language over PostgreSQL and introduces operators 
to select events from a stream using a sliding window. The STREAM project [26] 
extends SQL to processing an event stream by introducing a sliding window 
specification and an optional sampling clause. For example, it was possible to 
make queries that refer to the previous ten entries. Cayuga [13] also uses an SQL 
like language to specify the queries. 

The Aurora project [2] defines a stream query algebra that permits streams to be 
split, joined, and aggregated. Aurora is capable of working with either streams 
of data or data from a store. The SASE event language is an alternative approach 
that does not draw on SQL [38], but defines events of interest in a given time 
window that match a given pattern. 

Amit [3] uses an XML-based approach to define a situation definition language. 
A situation is a condition within which an event becomes significant and can be 
defined over a lifespan. Borealis [1] provides a mechanism to reapply queries on 
modified data; this might occur when operators realize that a correction is 
required to the stream and hence need to recompute the events. 

2. Graph-based approaches—In these approaches, the problem of event pro- 
cessing is considered as a network of a loop-free graphs and the user is allowed 
to specify their intent using the graph. Aurora [2] and the SPADE project [18] 
are examples of such systems. 
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Apache Storm [7, 24] models the processing as a topology. The input streams 
are modeled as spouts and the processing nodes are modeled as bolts. Twitter 
uses storm to process incoming tweets [6]. 

3. Programming-based approaches—Yahoo’s S4 stream processing engine [27] 

is inspired by Hadoop and process event tuples across a distributed set of 
processing elements (PE); each PE defines a processEvent() method that is used 
to process a particular event and in case another event is generated, it invokes 
the output() method. 
The map-update programming paradigm [23] is influenced by the MapReduce 
model and seeks to ease the task of event processing by breaking down the 
processing of events into the Map and the Update step and process them over a 
distributed set of systems. Since events are instantaneous, any state information 
between events is summarized into a context parameter to the Update step 
known as the slate. 

4. Advanced analytics—Most of the complex event processing frameworks focus 
on providing an infrastructure to handle large number of events. Additional 
processing to derive insight from the streams must be built on top [28]. Samoa 
[25] provides support for a pluggable data mining and machine learning library 
built over S4 and Storm. 


For our manufacturing intelligence application, we would want to specify all 
queries to include all pressure events within a time period to reach a peak value in 
the permissible range. 


6 Performance/Scalability 


The event processing system must be capable of handling events with a low latency. 
Twitter, for example, must be capable of processing 100000 tweets a minute [14]. 
In order to process a large number of events, a complex event system must [34] 


1. Avoid storage operations in the critical path: Storage operations in the critical 
path lead to additional latency in processing. Aurora [2] avoids all store oper- 
ations; instead store is performed optionally. This is the approach taken by 
Apache Storm [7]. 

2. Distribute operations: Gain performance through distribution of workload 
across multiple nodes. The system must be capable of responding to additional 
workload by scaling across multiple nodes [7]. The Medusa project provides for 
distributed naming and catalogs to provide for distributed operations [9]. It also 
provides mechanism for multiplexing event streams on the TCP layer. Given 
that events are transferred between processing nodes/brokers that are distributed, 
the queue sizes in between the nodes are critical to performance. The STREAM 
project provides a mechanism to optimize the internode queue sizes [26]. 
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3. Optimize query processing and scheduling: Queries need to be scheduled on 
the processing elements to ensure that QoS guarantees are met. The STREAM 
project [26] provides a structure to the query plans, thus enabling query plan 
sharing and optimization. It further exploits constraints on data streams to 
reduce memory overhead during computation. 


7 Fault Tolerance 


As streams of events are being processed by a distributed set of nodes, there is 
possibility that either some of the events may not reach or may be delayed in 
reaching the processing node. Further, it is also possible that some of the nodes that 
comprise the distributed system may fail. These are handled as 


1. Delayed arrival of events: Most complex event processing systems work on a 
sliding window waiting for events [2]. D-stream or discretized streams [39] is 
alternative approach that processes discretized portions of the input stream to 
account for lag in the input data. The computation itself is then performed using 
Spark [40] that uses resilient distributed datasets to ensure fault tolerance. 

2. Failing nodes: In Muppet [23], each node keeps track of all other nodes in the 
system. When a node is unable to send data to another node, it assumes that 
node is not functional and reports failure to the master node. The master node in 
turn broadcasts the failure status to all the nodes that then update their list of 
functional nodes. The work is then taken over by another functional node in the 
system. 


Having briefly discussed various features that must constitute a complex event 
processing system, we delve in more depth in the modeling of complex events. 


8 Modeling Complex Events 


In this section, we describe some of the important methods that have been proposed 
for modeling complex events. This would enable us to model the relationship 
amongst various events and the parameters of interest; for example, in the manu- 
facturing intelligence solution, this would allow us to express the relationship 
between various sensors and data values read from log files to the quality of the 
product. The first question is to enumerate the aspects of an event that are 
important. Next, we survey some of the different approaches that have been pro- 
posed for modeling events. 
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8.1 Event Aspects 


Generally, there are many attributes associated with an event. For a complete 
description of an event, it is necessary to store these attributes. Events commonly 
occur to one or more objects, which are considered participants in the event. Nor- 
mally, there is also a context associated with events. This context may be both spatial 
and temporal. The context need not be absolute—e.g., the temporal context may be a 
specified time interval after a previous event. The objects participating in an event 
may have a relationship—e.g., in a cricket game, there will be a batsman and a bowler. 
Events may also have a hierarchical structure and be nested inside one another. 
Figures 5 and 6 [37] list many of the important aspects of events. These include 


e The temporal aspect of the event, i.e., the time interval over which the event 
occurred. The time may be the physical or absolute time, relative to other events 
or logical (domain-specific time) such as the frame number in a video. 

e The causal aspect of events which links events in a chain of cause and effect is 
important when analyzing the reasons for occurrence of an event. For example, 
in studying a sports video, the causal linkages between different events in the 
game could be important. 

e The spatial aspect of the event encapsulates where the event occurred. As in the 
temporal aspect, the spatial aspect may be the physical location, or relative to 
other locations, or logical (e.g., the home of an individual). 

e In multimedia systems, the experiential aspects may be important. For example, 
in instant replay in sports videos, the analyst is seeking to replay the event in 
order to gain more insight. The media associated with the event is then 
important. 

e The informational aspect of the event consists of data about event such as the 
objects participating in the event, and the type of event (e.g., runs scored in a 
cricket game). 

e Events may be made up of subevents, and these are captured in the structural 
aspects of an event. 


Fig. 5 Design aspects of 
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Fig. 6 Aspects of an event 


8.2 Categories of Temporal Relationships 


In the discussion of event attributes above, we have listed the common types of 
temporal attributes. Due to having temporal attributes, multiple events can also be 
linked together in temporal relationships, such as one event occurring before 
another. Many relationships proposed in different event models were surveyed in 
[8] where it was shown that these relationships can be divided into two categories— 
timestamping and evolution constraints. The paper also contains a discussion of 
mathematical properties of models with these relationships. 


8.2.1 Timestamping 


Timestamping allows distinguishing between event model elements that are inde- 
pendent of time, and those that are time dependent. Elements to which times- 
tamping does not apply are called atemporal. Timestamping could be applied to 
entities, attributes, or relationships. An example of timestamping applied to an 
entity is the lifetime of a machine in a factory. Since the salary of an employee 
varies with time, timestamping can be applied to an attribute such as employee 
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salary. An example of a timestamped relationship is an employee—manager rela- 
tionship in a company, which is valid during the time a particular employee has a 
particular manager. 


8.2.2 Evolution Constraints 


Evolution constraints govern the evolution of the system being modeled and specify 
the permissible changes in elements of the model. By specifying evolution con- 
straints, the integrity of the database can be maintained. Some of the important 
changes include 


e Object Migration: an object of one class (e.g., employee) becomes an object of 
another class (e.g., manager). However, migration to director class may be 
disallowed. 

e Status Class: here, a timestamp is associated with the object migration, so that 
during its lifecycle, the object may be said to be in different statuses at different 
times. 

e Generation Relationship: an object of one class may generate an object of a 
possibly different class. An example is the situation where a manager in a 
business generates a project. 


8.3 Survey of Event Models 


In the following section, we survey some of the different models that have been 
proposed by different researchers. These models and their relationships are as 
follows: 


e Extensions to Relational Models: Relational databases are widely used as 
storage systems. Hence, the first model described is the TimeER model, which 
extends the widely used extended entity relationship model for modeling events. 

e Graph Models: It can be seen that the relationship between different aspects of 
an event can naturally be modeled by a graph. Therefore, the next model 
considered is Æ“, which proposes a graph-based model of events based upon 
RDF. 

e Extended Graph Models: Graph models such as E’ are designed for describing 
objective events such as events in a surveillance video or in a computer system. 
Generally, the models are also domain-specific. These models have been 
extended in different ways by various researchers. Examples include 


e F: The F model tries to simplify the construction of graph models for events 
by providing templates for simple events. These simple events can be 
combined to define complex events. 
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e SEM: The SEM model tries to account for historical or fictional events 
where there can be differing perspectives and interpretations of the event. 
The interpretation is explicitly modeled in the event model. 

e EventWeb: The EventWeb model is a two-level model where the top level is 
independent of the domain being modeled, and the lower level is 
domain-dependent. 


8.3.1 Entity Relationship-Based Model for Events 


The extended entity relationship model (EER) is a well-known technique for 
modeling data in a domain. Subsequently, the EER model can automatically be 
converted into a relational schema [15]. The ER model was extended to capture 
temporal aspects in [19]. Subsequently, methods for imposing temporal constraints 
were developed in [11]. The same technique can be used to map temporal ER 
constraints to NoSQL databases as well. Due to the importance of relational and 
NoSQL databases, we describe these techniques below. The discussion is based 
upon the approach in [11]. 


8.3.2 TimeER Model 


The TimeER model extends the EER model by adding timestamping information. 
Four types of temporal timestamping information are supported. These are 


e Valid time: the time interval during which a fact is true—for example, the time 
when an employee is a member of a department. 

e Lifespan: the time during which an entity exists in the model (not the database). 
For example, if the database models events in a company, the lifespan of an 
employee could be the time the employee is in the company. 

e User-defined time: these are not generic, like the above two times, but depend 
upon the semantics of the entity. For example, birthdays could be associated 
with an employee, but not a department. 

e Transaction time: the time when a fact is in a database. 


The concepts behind the TimeER model can be illustrated in Fig. 7, which 
describes part of the model needed for the manufacturing use case described in 
Sect. 1. Assume that enough data about the manufacturing process should be 
captured so that it is possible to find patterns of irregularity in the process that leads 
to defects. Each Wafer is identified by a Waferld and can have a Defect.' The 
Waferld has a lifespan timestamp indicating the time the wafer was in the system. 
The Defect also has a lifespan timestamp indicating when the defect was detected, 


'For simplicity in modelling, assume that each wafer can have at most one defect. The model can 
be easily extended to multiple defects. 
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Fig. 7 TimeER model for the manufacturing process 


and which may be of use in determining the cause. These timestamps are indicated 
by the annotation LS inside the corresponding boxes. The Wafer belongs to a Batch 
which has a BatchId. The Batch has a lifespan, but the Belongs to relationship has 
been modeled as a permanent relationship, in the same way that a parent-child 
relation can be considered to be permanent. The Batch can be immersed in a Tank 
which has a Tankld. Since the Immersed in relationship holds only during a certain 
time interval, it has a valid time timestamp, indicated by the VT annotation. During 
the immersion, the wafer is sprayed with a pressure that varies with time. This is 
represented by sampling the pressure at two different times during the spraying 
process. This is modeled as the attributes Pressure] and Pressure2 of the Immersed 
in relationship. Each attribute has a user-defined timestamp indicated by the UT 
annotation. Finally, the tank also has to be replenished periodically. This is indi- 
cated by the Replenished by relationship which, similar to the Immersed in rela- 
tionship, has a valid time timestamp. 


8.3.3 Mapping the TimeER Model to a Schema 


The above ER model can be mapped into a relational schema. A detailed 
description can be found in [15]. However, we outline the basics of the process 
below. 


e Mapping entities: a table is created for each entity. The entity attributes become 
the fields of the table, and one of the attributes is selected as the primary key. 
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e Mapping weak entity sets: a table is created for each entity set, with attributes as 
keys. The primary key of the identifying entity set is also added. The primary 
key for the table is the primary key of the identifying entity set together with the 
identifying key of the weak entity set. 

e Mapping relationships: a table is created for each relationship, with the primary 
key of each entity in the relationship as the fields. Additional fields could 
contain the attributes of the relationship (if any). The primary key is the com- 
posite primary key of all participating entities. 


A Wafer table has to be created for wafers. The table will have the Waferld and 
Defect as fields. Additionally, it is necessary to store the lifespan for the Wafer and 
the Defect. Lifespans can be represented by a start time and an end time. Thus, there 
would be start times and end times for the Wafer and the Defect. The Belongs to 
relationship can be stored as a table with the Waferld and Batchld as fields, since 
these are the entities participating in the relationship. The Jmmersed in relationship 
can be stored as a table with BatchId, Tankld, Pressurel, and Pressure? as fields. 
The valid time timestamp of the relationship, which represents the time the batch 
was immersed in the tank, can be represented by adding the start and end of the 
immersion times as fields to the Immersed in table. The user-defined time times- 
tamps for Pressure! and Pressure 2 can be stored as additional fields which record 
the times at which the pressure was recorded. The rest of the TimeER diagram can 
be similarly converted into a relational schema. 


8.3.4 Adding Temporal Constraints to TimeER 


Temporal constraints provide constraints on the evolution of the system. Some of 
the temporal constraints from [11] are listed below. For full details of the con- 
straints, we refer the reader to the original paper. Temporal constraints can apply to 
keys, attributes, or superclass/subclass relationships as below. 


e Time-invariant key: constrains the key for the entity to be unchanging during the 
valid time of the entity. 

e Temporal key: specifies the constraint that if two entities have the same key, 
they are the same entity. The difference between this and the time-invariant key 
is that a temporal key may change with time, but at any given time uniquely 
references only one entity; whereas a time-invariant key does not change with 
time. 

Time-invariant attribute: constrains the attribute to not change with time. 
Temporally disjoint subclass/superclass: constrains an entity to be part of only 
one subclass. 

e Time-invariant subclass/superclass: specifies the constraint that the entity’s 
subclass membership cannot change with time. 


Figure 8 shows how some of the above constraints can be applied to the TimeER 
diagram for the manufacturing process. The notation TI has been added to the 
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Waferld, Batchld, Solutionld, and Tankld. This indicates that those fields are 
time-invariant keys. Note that in the current model, the Solution/d is time-invariant, 
so keep track only of the solution type. If it is desired to keep track of the solution 
batch number (1.e., it is assumed that different batches of the solution are used for 
spraying) then the SolutionId could be modeled as a temporal key. These con- 
straints can be enforced by the database. 

The TimeER diagram has been modified slightly to enforce the constraint that 
when the solution in a tank is being refreshed, no batch can be immersed in the 
tank. The Tank superclass has been split into two subclasses—Replenishing Tanks 
and Replenished Tanks. Replenishing Tanks are those tanks where the solution is 
being replenished. Batches can only have a Immersed in relationship with 
Replenished Tanks, not Replenishing Tanks. The notation d(T) indicates that the set 
of Replenished Tanks and Replenishing Tanks have a temporally disjoint 
subclass/superclass relationship, i.e., that batches cannot be immersed in tanks 
where the solution is being replenished. This constraint can again be enforced by 
the database. 
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Fig. 8 TimeER model for the manufacturing process with temporal constraints 
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8.4 Graph-Based Models for Events 


The example given above maps events into relational databases. However, since 
relationships are extremely important in event modeling, many researchers have 
proposed graph models of events. Graph models of events are useful when mod- 
eling events with many relationships, and where the relationships are not known in 
advance. 

In the following we describe E*, a graph model of events proposed in [20]. This 
is an extension of the E model proposed in [36]. The temporal aspects of E* uses 
the DOLCE ontology defined in [17]. The spatial aspects of E* are derived using 
the approach proposed in [29] and is described in a later subsection. The following 
subsection describes DOLCE. The temporal aspects of E* are described 
subsequently. 


8.4.1 DOLCE Ontology 


The DOLCE ontology is intended to reflect the categories that underlie natural 
language, and is intended to have a cognitive bias. The need for DOLCE arises 
from the need for a foundational ontology that can serve as a basis for negotiating 
meaning between different specialized ontologies that may be developed for 
extracting semantics from objects in the Web. It is likely that the structure and 
relationship underlying videos and other documents on the Web reflect these 
underlying cognitive concepts. Since these concepts are already incorporated into 
DOLCE, the DOLCE ontology can be a useful tool for this purpose [17]. 

Figure 9 illustrates many of the important categories in the DOLCE ontology. 


e Entities are divided into Abstract, Endurant, and Perdurant categories. Infor- 
mally, it can be stated that endurants are objects or attributes (called Qualities in 
the DOLCE ontology) while perdurants are occurrences. Qualities are classed as 
endurants since this appears to be the way in which they are treated in natural 
language [17]. For example, in the sentence “The machine’s temperature is 
increasing,’ temperature is treated as an object whose value is changing (in the 
DOLCE ontology, the value of a quality is called a gualia). Endurants partic- 
ipate in perdurants, e.g., in the manufacturing use case, batches of wafers are 
dipped in a solution. 

e As stated above, endurants are divided into qualities and substantials, which 
represent objects. Qualities and substantials are further subdivided as shown in 
the diagram. It can be seen that space and time are treated as subclasses of 
quality in the DOLCE ontology. Substantials can be regarded as relatively 
unchanging aggregates of qualities in the DOLCE ontology. 

e Perdurants or occurrences can be divided into events and statives. Stative per- 
durants have the property that the composition of any two stative perdurants of a 


Complex Event Processing in Big Data Systems 


153 


ArT 
inal 
AB ED PDO 
Abstract Endurant Perdurant/ 
f \ | f k 
Fact Set R ' \ 
a eS f: y 
Temporal Physical Non-physical  / Event Stativa 
Region Region Region / N ' 
» 7 XY Q SB rd \ 
gF Quali Substantial y r 
Time Space pe ny F ol ACH Acc 
Interval Region 7| Fi a Achievement Accomplishment 
NS 
\ 
“ee ST PRO 
TO N 3 ` State Process 
Physical Non-physical 
Substantial substantial 
AG F POB XPOE 
Aggregate Feature Physical Non-physical 
F j q | Fi d ‘ " i 7 ` n 
uv ac RP FL APO NAPO MOB SOB 
Amountof Arbtrary Relevant Place Agentve Non-agentve Mental Object social Object 
Matter Collection Part Physical Physical Fa k 
Object Object ff 
Pd ù 
Fa w 
ASO NASO 
Agentive Non-agentive 
Social Object Social Object 
SAG sc 
Social Agent Society 


Fig. 9 DOLCE upper level categories 


particular type s is also a perdurant of type s. John is sitting is an example of a 
stative perdurant because the composition of any two (consecutive) perdurants 
of type John is sitting is another perdurant of type sitting. Climbing Mount 
Everest, on the other hand, is not a stative perdurant but an eventive perdurant, 
since the composition of any two perdurants of type climbing Mount Everest is 
not a perdurant of type climbing Mount Everest. 
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e Stative perdurants can be divided into states and processes. A perdurant is 
classified as a state if every part of the perdurant satisfies the description. For 
example, John is sitting 1s classified as a state, since John 1s sitting in every part 
of the event. This property is referred to as the homeomeric property. However, 
John is speaking is classified as a process, not a state, since there could be short 
pauses in between the times when John is speaking. 

e Events can be subclassified into achievements (if they are atomic) or accom- 
plishments (if they are compound events with parts). 


8.4.2 E* Model 


Figure 10 shows part of an E* model for the manufacturing use case. The model 
fragment models the batching of wafers into batches. This is modeled as an 
achievement, since it is an atomic event. The node Ach::01 in the RDF graph 
denotes this. The makeBatch predicate indicates that the event Ach::0/ is to make a 
batch out of the blank RDF note _wb, which is an ordered list consisting of a Wafer 
and a Batch. The Wafer has attributes Waferld and Defect as before, and the Batch 
has an attribute BatchId. The time taken for the event is given by the timelnterval 
predicate, which has a startTime and an endTime. In terms of the DOLCE ontology, 
Wafer and Batch are non-agentive physical objects, and Waferld, Defect, Batchld, 
startTime, and endTime are qualities. In terms of the implementation, since E* 
model is in terms of an RDF graph, the simplest alternative is to use an RDF 
database for storage and querying. More advanced methods are given in [20]. 
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Fig. 10 RDF graph for E* model for the manufacturing process 
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8.4.3 F 


An alternative event model based upon composition of fundamental event patterns 
is proposed in [31]. The basic idea in F is that complex events can be composed by 
combining the fundamental event patterns. An RDF/OWL implementation of F is 
available at [30]. The fundamental event patterns of F are 


Participation: indicating that an object is participating in an event 
Temporal duration of events: the time interval over which an event occurs 
Spatial extension of objects: the space occupied by the event 

Structural relationships: three types of structural relationships are modeled 


e Mereological: indicating that certain events are subevents of others 
e Causal: indicating some events cause others 
e Correlation: indicating that there is a correlation between events 


e Documentary support: the media or other documents in which the events are 
recorded 

e Event interpretations: an interesting feature of the F model is the ability to 
model an interpretation of an event, since different observers may disagree about 
the causal relationships or participants in an event. 


Figure 11 shows example F patterns as well as their usage. Figure 11 shows the 
participation pattern. It can be seen that the pattern includes all the components of 
an object participating in an event, such as the type of object, its qualities, and the 
type of event and its qualities. Similar patterns are defined for all the fundamental 
patterns described above. To model an event, it is stmply necessary to stitch the 
required patterns together, omitting any null objects or qualities. Figure 12 shows 
part of the model of the situation where an electric pole has snapped, and has led to 





Fig. 11 F participation pattern 
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Fig. 12 Combination of F patterns 


a power outage. The bottom part of Fig. 12 is derived from the participation pattern, 
and lists the objects participating in the event—the house and a citizen. The top part 
is drawn from the causality pattern and shows the snapped telephone pole as the 
cause of the power outage. 


8.4.4 The SEM Model 


The models described earlier in this section try to specify precisely the semantics of 
the various entities in the model. However, in the World Wide Web, it is likely that 
the information sources would be inconsistent, and that different information 
sources would model the same information in a slightly different manner [4]. It is 
also possible that different information sources may contradict each other due to 
differences in opinion. 

The Simple Event Model (SEM) introduced by [35] is a model that attempts to 
cater to the above requirements. A novel feature of SEM is that in order to allow 
integration of information from multiple sources, it puts few restrictions on the 
individual elements of the model. For example, it does not distinguish between 
instances and types, and does not require individuals to be disjoint from places. 
Places need not be specified (as conventional in many models) by a latitude and 
longitude; they may be fictional, to allow for specification of fictional events. In 
addition, SEM has a Prolog API which makes it simpler to build event models. 
Also, the range and domain of any RDF statement in the SEM model is always 
unrestricted Figure 13. 
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Fig. 13 SEM model for historical statement 


There are four core classes in the SEM model. As stated earlier, these need not 
be disjoint, so that an actor could also be a place. The core classes are 


sem:Event: the class that models events 
sem:Actor: the entities involved in the event 
sem:Place: the place where the event occurred 
sem:Time: the time when the event occurred 


Additionally, the SEM model also allows specification of property constraints, to 
capture the constraints present in the event. The role of constraints will be described 
by considering the SEM model for the historical statement “The Dutch launched the 
first police action in the Dutch East Indies in 1947; the Dutch presented themselves 
as liberators but were seen as occupiers by the Indonesian people” [35]. It can be 
noted that the statement expresses two different points of view—the Dutch, who 
saw themselves as liberators, and the Indonesians, who saw the Dutch as occupiers. 
These opposing views are modeled by the blank RDF node that is the object of the 
sem:hasActor predicate (in the top center of Fig. 9). This node points to two dif- 
ferent sem:roleType nodes. In the top branch, sem:roleType leads to an occupier 
value, which is also connected by an sem:accordingTo predicate to a dbpedia: 
Indonesia. This sub-graph captures the viewpoint that according to the Indonesians, 
the role of the Dutch was that of occupiers. The dbpedia:Indonesia value is of type 
sem:Authority, which designates a source of data that may conflict with a different 
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source. A similar sub-graph starting with the bottom branch of the sem:roleType 
predicate indicates that according to the Dutch (dbpedia:Netherlands), they were 
liberators. Details can be found in [35]. 


8.4.5 EventWeb Model 


A two-level event model, consisting of an upper level domain-independent ontology, 
and a lower level domain-specific model, was proposed in [29, 32]. The upper level 
ontology contains continuants, which are time-invariant and keep their identity 
through change; and occurrents, which model events or processes. The class 
Dynamic_Entity models continuants that have dynamic spatial behavior. Temporal 
relationships are indicated using Temporal RDF proposed in [21]. In temporal RDF, 
the validity of an RDF tuple is indicated by adding the time range in which it is valid 
within square brackets to the predicate. For example, in Fig. 14, the Military_Unit 
participates in a Military_Event in the time intervals t, and te, This is indicated by the 
RDF tuple (Military_Unit, participates_in, Military_Event) [t,, te] where the quan- 
tity in square brackets indicates the valid time of the RDF tuple. In the RDF graph, 
the notation [t,, te] after the participates_in predicate similarly indicates the valid 
time of the predicate. Spatial entities and relationships are based upon a GIS 
ontology, and contains objects for specifying geographical regions. Relationships 
available in the GIS ontology include concepts such as inside. 

It can therefore be seen that the EventWeb lower level model consists of three 
parts—(i) a temporal model based upon temporal RDF, (1i) a spatial model based 
upon a GIS ontology, and (i111) a domain-dependent model (called a theme). Entities 


Upper-level ontology rdis: subClassOf —————_—_—+ 
rdis: subClassOf (used for integration) 


_ C Occurent >, rdk:Property — mme —+ 





| located, _at: lr, a) ae 
<> a> =>. ae: 
occured e 


] 1. F š 
i 
i a G i 








ls gives: [t.. 


traing_at:([t,.t,] 





| assigned_to: } 

` [t,. A ~~ 1 eases icipates_ in: 
‘\on_crew_of: [t e] Cima Une [t tl 
Gap. used_in:(t,. t] _ ees 


Fig. 14 EventWeb event model 
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in these three lower level models are mapped to corresponding entities in the upper 
level ontological model. It can be seen that the EventWeb model is therefore easily 
generalized to different domains. 


9 Conclusions 


We have outlined an example of a manufacturing intelligence process and illus- 
trated how modeling it as a complex event model can lead to more structured 
approach toward monitoring and identifying potential issues in the process. Such 
formal approaches can also help in identifying root causes and potential impact of 
various parameters on the manufacturing process. 
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Unwanted Traffic Identification 
in Large-Scale University Networks: 
A Case Study 


Chittaranjan Hota, Pratik Narang and Jagan Mohan Reddy 


Abstract To mitigate the malicious impact of P2P traffic on University networks, 
in this article the authors have proposed the design of payload-oblivious privacy- 
preserving P2P traffic detectors. The proposed detectors do not rely on payload sig- 
natures, and hence, are resilient to P2P client and protocol changes—a phenom- 
enon which is now becoming increasingly frequent with newer, more popular P2P 
clients/protocols. The article also discusses newer designs to accurately distinguish 
P2P botnets from benign P2P applications. The datasets gathered from the testbed 
and other sources range from Gigabytes to Terabytes containing both unstructured 
and structured data assimilated through running of various applications within the 
University network. The approaches proposed in this article describe novel ways to 
handle large amounts of data that is collected at unprecedented scale in authors’ 
University network. 


1 Introduction 


In recent times, Computer Networks are fast evolving, with the number of users, 
applications, computing devices, and network protocols increasing exponentially. 
However, fundamental challenges still remain, such as scalability, security, and man- 
agement issues. Today, P2P applications draw a significant attention from network 
researchers. P2P file-sharing software is often a primary source of information leak- 
age, which has become a serious concern for University network users, and other 
public organizations. To detect and prevent information leakage from an internal 
node or user’s machine, network administrators must deploy appliances that han- 
dle these types of applications in an intelligent manner while providing assurance of 
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security and privacy to the end users of these applications. BitTorrent, and other P2P 
file-sharing networks, allow people to share files amongst themselves. Sharing files 
on ones’ computer with unknown users (peers) on a public Internet by having logical 
identities brings in natural concerns related to security and privacy. P2P applications 
open up a TCP or UDP port at the network appliance to get connected to the other 
side of the application (peer). Hence, once a port is opened, the peer is no longer 
protected from malicious traffic coming through it. Another major security concern 
while using a P2P application is how to determine the authenticity of the content that 
is downloaded or accessed. In other words how can you make sure that you have not 
received malicious content in the form of spyware or a bot. The kind of impact these 
P2P applications generate on the performance of university campus networks have 
not been well studied. 

As P2P networks can control their own behavior bypassing the scrutiny from a 
centralized administrator, it is impossible to implement security protections on a 
deployed P2P network. Moreover, amongst many peers participating in the applica- 
tions, it is hard to identify who are those peers who are good and who are bad. Mali- 
cious users can actively play the role of insiders. In this paper authors have investi- 
gated the negative impacts of P2P traffic on network security and developed privacy- 
preserving P2P detectors to mitigate these negative impacts. In particular, authors 
have used their University network to collect and mine comprehensive datasets con- 
taining a wide variety of applications like Web, P2P, SMTP, Streaming, etc. The 
dataset collected also contains regular P2P (benign) and malicious P2P traffic from 
dormitories, and academic VLANs situated across the campus. Authors have evalu- 
ated these datasets quantitatively using existing techniques to reveal how P2P traffic 
affects the accuracy, sensitivity, and detection capabilities of network security appli- 
ances. 

This case study of identification of unwanted traffic in large-scale University net- 
works is based on the network of the authors’ University. This work provides an eval- 
uation of the impact generated by P2P traffic on University networks, and presents 
various approaches (existing in literature as well as from authors’ prior research) 
which are used to tackle this kind of unwanted traffic in large-scale University net- 
works. 

Our University network serves five thousand to six thousand on-campus students 
along with faculty and guest users every day, providing wired as well as wireless 
connectivity in the different parts of the University campus. Internet connectivity is 
provided in the University by leased lines from two Internet Service Providers (ISPs). 
The University network consists of a three-tier architecture, consisting of two Core 
switches (one for fall-back), and multiple Distribution switches and Access switches 
with a 10 Gb backbone fiber. Wireless connectivity is also provided in the academic 
blocks and other selected areas within the campus by deploying Wi-Fi access points. 
Access to different Internet-based services hosted by the University is provided in a 
‘DMZ’ (demilitarized zone). The DMZ hosts the Web (HTTP(S)) service, FTP ser- 
vice, and the Institutional email service (SMTP) of the University. The entire archi- 
tecture is shown in Figs. 1 and 2. Figure 1 gives the network architecture with respect 
to the network design of the University, wherein two Core switches and their con- 
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University. 
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The network is protected by a commercial Unified Threat Management (UTM) 
system which controls the ingress and egress Internet traffic. The UTM performs the 
task of traditional Intrusion Detection/Prevention Systems (IDS/IPS). Along with 
this, it also provides the functionalities of antivirus, anti-malware, and spam filtering. 
It also performs URL filtering and blocks access to potentially suspicious websites 
hosting malware, spyware, adware, etc. 

Owing to a large number as well as variety of users in a large-scale University, 
such networks generate a huge amount of data every day. Web-based traffic related 
to web browsing, social networking, search engines, and education-related mate- 
rial forms the major content of Internet traffic. P2P-based applications—although 
popular amongst students—are notorious for consuming large amounts of Internet 
bandwidth. Although these applications are blocked by the UTM, we observed that 
students are still able to evade the UTM and run many such P2P applications on the 
University network. This might be attributed to the fact that many such applications 
use encryption (and thus cannot be detected by signatures) and randomize their port 
numbers (and thus cannot be filtered based on port numbers). 

At an average, the network traffic generated at our University is around one Ter- 
abyte per day. A snapshot of the distribution of different types of applications run- 
ning in the network, as captured by the UTM, is given in Fig.3. As is evident 
from the chart, the majority of the traffic generated in the University is web based 
(HTTP/HTTPS). 

The UTM records the web activity of the users of the network, also performs 
attack prevention and threat monitoring. Figures 4 and 5 give logs of IPS events and 
Web usage events (respectively) generated by the UTM for a period of one month (20 
July 2014 to 20 August 2014). IPS events are those events which refer to intrusion 
activities identified by the UTM device. Web usage events are those events which 
are recorded by the UTM device as web activities carried out by several users using 
protocols like HTTP(S), TCP, FTP, etc. 


Fig. 3 Distribution of Distribution of protocols 
protocols 





e https (60%) = http (30%) = others (dns, pop3, imaps) (10%) 
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Fig. 4 IPS events 


2 Early Challenges in P2P Networks 


Any decentralized and distributed computing environment naturally involves many 
challenges of security, and P2P overlay networks are no exception. In this section, we 
will review some of the prominent attacks and security threats which were prevalent 
in P2P networks in their early days (the first decade of the twenty-first century). 
However, this list is not exhaustive. For a taxonomy of attacks on P2P overlays, we 
refer the reader to [48, 50]. 
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Fig. 5 Web usage events 


2.1 Application Level Attacks 


2.1.1 Index Poisoning Attacks 


P2P file-sharing involves the use of ‘indices’, which are used by peers to search and 
find locations of files desired by them. If an attacker inserts massive numbers of 
bogus records into the index for a set of targeted titles, it is called as an index poi- 
soning attack [25]. When a legitimate peer searches for a targeted title, the index will 
return bogus results, such as bogus file identifiers, bogus IP addresses, etc. 

Key defense against index poisoning is to provide a verifiable identity. This may 
be done through Public Key Infrastructure (PKI) cryptography, as in [2]. However, 
PKI requires the presence of a trusted certificate authority, which is not always feasi- 
ble in a P2P network. Even if a certificate authority is implemented, it will face huge 
workload due to a large number of peers as well as high peer-churn. 
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2.1.2 Query Flooding 


Flooding of search queries is basic search mechanism employed in many P2P net- 
works. A peer looking for a resource will broadcast its query to its immediate neigh- 
bors. If a neighbor does not have the resource, the query is further forwarded to its 
neighbors. This goes on until the ‘hop limit’ is reached or the resource is found. 
Malicious peers can exploit this flooding mechanism to generate a query flooding 
attack. A malicious user may generate as many useless queries as possible to flood 
the network. As a result, the resources of the network will be engaged in serving 
these requests, and benign users may face service degradation. 

A solution for preventing query flooding in Gnutella was implemented by putting 
an upper limit on the number of queries a node can accept from a requesting peer 
[8]. After this number is reached, all other queries are dropped. 


2.1.3 Pollution Attacks 


Since peers participate in the P2P network using virtual identities, it is easy for an 
attacker to spread unusable or harmful content without getting the risk of getting 
caught. Peers are engaged in downloading chunks of files from different sources. 
An attacker may host a file x with himself, but replace the original contents with 
some junk pieces of data or some malware. When a peer, looking for file x, down- 
loads the content from the attacker, he receives ‘polluted’ content in the form of 
a corrupted or malicious file. Hash verification and chunk signing are some of the 
common approaches used against pollution of content in P2P networks [10, 24]. 


2.1.4 Rational Attacks 


P2P networks are based on the principle of peers cooperating to mobilize and share 
resources. If peers are unwilling to cooperate, the P2P network will fail to survive. A 
self-interested node, however, may attempt to maximize the benefits it receives from 
the network while minimizing the contributions it makes. This has been described 
as a rational attack. A “free-rider’ [13] is a term used in P2P systems to describe 
a peer who utilizes the services and benefit of the network, but does not make any 
contributions. Since P2P networks are inherently about collaboration and sharing of 
resources, the services of the network face degradation if most of the users behave 
in a Selfish way and choose to free-ride. 


2.2 Identity Assignment Attacks 


Peers participate in P2P networks by assuming virtual identities. As a matter of fair- 
ness, it is assumed that one physical entity shall own one random virtual identity in 
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the P2P network by which it will participate in network activity. However, a mali- 
cious peer may attack this identity assignment principle by creating multiple identi- 
ties referred to as ‘sybil’ identities [11]. By positioning its ‘sybil’ nodes at strategic 
positions within the network, an attacking peer can try to gain illegitimate control 
over the network or its part. Sybils can cause multiple damages. By monitoring the 
traffic flowing through the sybil nodes, the attacker can observe the traffic patterns 
of other peers whose traffic is flowing through them, and can also attempt to misuse 
the communication protocol in other ways. Sybil nodes may also be used to forward 
routing or search requests to other malicious/sybil nodes, thereby disrupting the nor- 
mal functioning of the network. The attacker can also gain control over certain files 
shared in the network and can choose to deny access to them or corrupt them. 

In DHT-based P2P networks, the nodes having the ID closest to the ID of a par- 
ticular file are responsible for maintaining information about that file. Hence, an 
attacker could possibly control the availability of a certain resource if he maps his 
IDs very close to that resource. Authors in [27] demonstrated the possibility of cor- 
rupting the information in the DHT-based Kad network by spreading polluted infor- 
mation through a ‘Node insertion’ or ‘ID mapping’ attack. The attacker may carry 
out this attack to pollute information about keywords, comments or list of sources. 
Authors in [44] implemented an attack for keywords where the attacker manipulates 
the ID assignment mechanism of Kad and generates an ID of his own, which lies 
very close to the targeted file keyword. Hence, when a search is carried out for that 
particular keyword, it does not yield correct results. Rather, the bogus information 
planted by the attacker is returned. 

A number of approaches have been suggested to defend P2P systems against sybil 
attacks. One of the early work in this regard is the use of computational puzzles [4]. 
Authors in [18] evaluate the use of CAPTCHAs in detecting sybils. An approach 
for detecting sybils using ‘psychometric tests’ is proposed in [19]. Authors in [49] 
have argued that a malicious user can create many identities, but only a few relation- 
ships of trust. They use this fact to locate disproportionately small ‘cuts’ in the graph 
between the sybil nodes and the honest nodes, and attempt to bound the number of 
identities a malicious user may create. 


2.3 Routing Attacks 


2.3.1 Route Table Poisoning Attacks 


Due to high rate of churn (joining and leaving of peers) in P2P networks, peers have 
to regularly update their routing tables in order to perform correct routing lookup. 
Peers create and update their routing tables by consulting other peers. If a malicious 
node sends false information in its routing table update, it will corrupt the rout- 
ing table entries of benign nodes, leading to queries being directed to inappropriate 
nodes or nonexistent nodes. 
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Different solutions have been developed to counter route table poisoning by 
imposing certain requirements on the participating peers. In the Pastry network [40], 
each entry in routing tables is preceded by a correct prefix, which cannot be repro- 
duced by a malicious entity. The CAN network [37] considers the round-trip time in 
order to favor lower latency paths in routing tables. Authors in [6] propose ‘induced’ 
churn as a counter against such attacks. 


2.3.2 Eclipse Attacks 


Each node in the P2P network maintains overlay links to a set of neighbor nodes, and 
each node uses these links to perform a lookup from its neighbors. Thus, an attacker 
may be able to control a significant part of overlay network by controlling a large 
part of the neighbors of legitimate nodes. This is known as an eclipse attack. Eclipse 
attacks are escalated forms of identity assignment attacks or route table poisoning 
attacks described above. If an attacker is able to generate a large number of fake 
identities and place those identities in the overlay network, he could mediate most 
overlay traffic, and thus eclipse legitimate nodes from each other. A pictorial repre- 
sentation of node insertion attacks and sybil attacks being escalated into an eclipse 
attack is given in Fig. 6. 

Authors in [45] describe the eclipsing of search keywords in Kad P2P network. 
Their experiment explains that an eclipse attack and a sybil attack can be performed 
quite similarly, except that the Kad ID space covered for an eclipse attack is much 
smaller. For eclipsing a certain keyword k in a DHT-based network such as Kad, the 
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Fig. 6 Node insertion attack and sybil attack escalated to an eclipse attack 
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authors in [45] choose to position a certain number of sybils as close as possible 
to the keyword k. Then, the sybil nodes are announced to the benign peers in the 
network which has the effect of ‘poisoning’ the regular peers’ routing tables for k and 
attract all the route requests for k to the sybil nodes. Authors in [45] claim through 
their experiments that as few as eight sybil peers were sufficient to ensure that any 
request for k terminates on one of the sybil nodes which were strategically planted 
in the network to lie as close as possible to k. Since all requests for k go via the sybil 
nodes, the attacker has effectively eclipsed the keyword k from all peers in the Kad 
network. 

The developer community of eMule (a P2P application which uses the Kad net- 
work) had responded to these security threats by making suitable changes to their 
applications. For example, the ‘change-log’ for eMule’s version 0.49a released in 
February 2008 (available at http://www.emule-project.net) states, “Kad will now 
enforce certain limits when adding new contacts to the routing table: No more than 1 
KadNode per IP, 2 similar KadNodes (same bin) from a /24 network and max 10 dif- 
ferent KadNodes from a /24 network are allowed”. Some countermeasures have also 
been proposed in literature, such as an optimized routing table and verified routing 
table [5], and a combination of induced churn and one-hop redirection [35]. Authors 
in [51] presented a scheme for generating node IDs which requires a user to solve 
a computational puzzle generated by their network parameters together with time- 
related information. The authors claim that such an ID generation mechanism makes 
ID assignment and eclipse attacks computationally infeasible for an attacker, while 
levying only a small overhead on a regular user. 

The implementation of many such countermeasures made large eclipse attacks 
very difficult or computationally prohibitive for an attacker. But, the emergence of 
P2P botnets provided a new ground for the adversaries to launch large-scale attacks 
without the time and effort required for an eclipse attack. P2P botnets can either be 
created on an existing P2P network’s protocol (for example, Storm used the Overnet 
P2P network [46]) or they may use a custom P2P protocol to build their own net- 
work of infected hosts (for example, Sality [12]). The bot master does not need to 
create multiple sybil identities or perform node insertions. A victim machine, upon 
infection from a malware or trojan, joins and becomes a part of the P2P network of 
the botnet. The decentralized architecture also helps in evading detection. 


3 Traditional Techniques 


Different University networks have different access control policies to access the 
Internet, and a network administrator must configure the Firewall accordingly. Many 
University networks may not deploy a commercial firewall to monitor the unknown 
traffic in each subnet. To accomplish task of detection of unwanted traffic, many 
open-source tools are available, such as Snort [39], Suricata [9], Bro [29], etc. In 
this section, we will discuss traditional approaches for P2P traffic detection using 
port-based, payload-based, and machine learning-based approaches. 
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3.1 Port-Based Classification 


The Internet Assigned Numbers Authority (IANA) has a list of registered port num- 
bers of a known application. Using these port numbers, an application communicates 
between two IP endpoints over TCP or UDP. IANA has defined some ‘well-known’ 
port numbers which are in the range 1—1024. Port-based classification technique is a 
traditional approach of classifying Internet traffic based on the port number being 
used by an application [42]. If a network administrator wants to block based on 
an application using this approach, many false positives (FP) and false negatives 
(FN) may be generated. This is explained as follows. First, some applications are 
not registered with IANA (e.g., certain P2P applications). Furthermore, certain P2P 
applications masquerade their traffic by using well-known port numbers used by web 
applications (port 80 and 443). “Torrent’ based applications have this property that 
their initial communication uses port 80 to contact the tracker server. Once the server 
replies with the peer list, the application uses some random port number to accom- 
plish the rest of the task. In certain cases, if IP layer security is enabled, it is almost 
impossible to even know the port numbers used. 


3.2 Payload-Based Classification 


Payload-based detection approach inspects the payload contents of each and every 
packet. ‘Signatures’ are extracted for different applications from their TCP/UDP/ 
HTTP payloads [43]. These signatures are employed by detection tools to and 
improve the accuracy of unwanted traffic detection. This approach is also known 
as Deep Packet Inspection (DPI). This technique can accurately classify the traffic at 
wirespeed. Although this technique is highly accurate, it requires significant amount 
of processing power. Furthermore, this approach proves less accurate on encrypted 
and proprietary protocols. Furthermore, the widespread commercial deployments 
of such DPI products has ensued a global debate on the ethical, legal and privacy 
implications of this practice. 

In our initial experiments, we used payload-based identification of P2P traffic 
using an open-source IDS Snort. To detect P2P traffic, we extracted signatures from 
multiple P2P applications from network traces obtained from our campus. A sample 
of these signatures is given in Table 1. 


3.3 Classification Based on Machine Learning 


The techniques described in previous sections depend upon inspection of the trans- 
port/application layer signature and port numbers. These approaches have certain 
limitations, as mentioned in the previous sections. Later, researchers have devised 
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Table 1 Application signatures 
Application TCP UDP 


DC++ $Send, $Connect, $Get, $Search, $SR, $Pin, $MyINFO, | — 
$MyNick, $Hello, $Quit, $ADCGET, $ADCSND, 
$Supports, $dcplusplus 


BitTorrent GET, /announce, info_hash=, peer_id=, event=, 13 z 
BitTorrent protocol 


GNUTella GNUTELLA, GNUTELLA CONNECT, GNUTELLA |- 
OK 


eDonkey Server 3A eMule — 





new approaches based on statistical characteristics to overcome the limitations of 
preceding approaches and accurately identify traffic application with low complex- 
ity, and ensure users’ privacy at the same time [21, 23]. The underlying idea is that 
the traffic has certain statistical patterns at network layer that are unique for differ- 
ent classes of applications (e.g., packet lengths, inter-arrival time of packets, bytes 
transferred per second, etc.). To deal with statistical patterns in multidimensional 
datasets generated from traffic flows, Machine Learning (ML), and pattern recogni- 
tion techniques were introduced in this field. ML is a powerful technique in the area 
of data mining which finds useful patterns from the given data. ML techniques are 
broadly categorized into two learning groups: Supervised learning and Unsupervised 
learning. 


4 Approaches for Identification of Unwanted Traffic 


4.1 Dataset 


Owing to their large number and variety of users, University-wide networks generate 
a large amount and variety of traffic. For the purpose of this case study, we obtained 
network logs from the backbone router of our University. Majority of this data is of 
web-based traffic. Apart from this, we also captured the data of popular file-sharing 
P2P applications which are commonly used by on-campus students. DC++ is a pop- 
ular P2P application used for files sharing on the network LAN. Certain Internet- 
based applications like BitTorrent, eMule are also popular amongst students. The 
details of this dataset are given in Table 2. 

Apart from benign Internet data, we also gathered data for malicious applications 
such as botnets. Part of this data was generated in a testbed at our University by run- 
ning the executables of these botnets in a controlled, virtualized environment. Part of 
it was obtained from other sources, namely University of Georgia [36], University of 
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Table 2 Benign dataset details 


Application No. of bytes 
HTTP ©) w o y y g 
SMTP/POP3 40 MB 

P2P apps (BitTorrent/eMule) 90 MB 


Table 3 Malicious dataset details 


Botnet name Type/size of data Source of data 


Sality Infects exe files, .pcap file/ 25 MB Generated on testbed 
disables security 
software 


Storm Email Spam .pcap file/ 4.8 GB University of Georgia 


Waledac Email spam, password | .pcap file/ 1.1 GB University of Georgia 
stealing 


Zeus Steals banking .pcap file/ 1 GB Univ. of Georgia, 
information by MITM Czech Technical 
key logging and form University + 
grabbing Generated on testbed 


Nugache .pcap file/ 58 MB University of Texas at 
Dallas 


Texas at Dallas [28] and Czech Technical University [15]. The details of this dataset 
are given in Table 3. 








4.2 Ensemble Learning Approach 


One of the promising approaches to identify unknown traffic at University backbone 
networks is by using ML techniques. Our approach is shown in Fig. 7. In this process 
we accumulate the packets into flows. We constructed ‘flows’ using the 5-tuple, 1.e., 
(SourcelP, DestinationIP, Sourceport, Destinationport, Protocol). 

Ensemble learning techniques combine multiple individual models for better pre- 
diction. The performance of our approach using ensemble learning techniques like 
Stacking and Voting is shown on Fig. 8. In this section, we first define Stacking and 
Voting, which are two of the most well-known ensemble approaches. Then, we dis- 
cuss our approach for identification of P2P traffic using ensemble learning [38]. 


e Stacking: This technique has three layers as shown in Fig. 8. In the first layer, it 
takes the training dataset as input. The second layer, the input dataset is fed to the 
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Fig. 7 Flow-based approach 
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Fig. 8 Ensemble learning 


several ‘base classifiers’ to obtain the individual predictions. In the third layer, a 
‘meta classifier’ is employed to obtained the final prediction. 

e Voting: This approach is similar to that of Stacking till the second layer. The only 
difference is that there is no meta classifier used in this technique. In the third layer, 
it combines the all probability estimations from the base classifiers and classifies 
an instance based on the majority vote. 
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Table 4 Ensemble classifiers 







Recall Accuracy 


‘Rell O O O 
ws we 9 [os wee 
w7 w» 7 [øm w» 





Stacking 


Our approach using ensemble learning: 


e Step I: We captured the network traffic at our core switch and stored them in SAN. 

e Step II: Once desired captures are obtained, we preprocess the traces and filter 
only IP packets. Then, we obtained desired flows to extract features. We extracted 
a total of 24 features from each bi-flows (1.e., a pair of unidirectional flows between 
the same 5-tuple) and stored them into comma separated values (CSV) format. 

e Step III: After feature extraction, we labeled each instance according to class as 
P2P or non-P2P for classification. 

e Step IV: We used ensemble learning technique to classify network traffic. In the 
offline phase, we experimented with Naive Bayes, Bayesian Networks, decision 
trees as base classifiers and random forest as decision classifier. In online phase, 
we Classified unknown samples (i.e., no class label is assigned to it) using model 
built from off-line phase. The experimental results obtained are shown in Table 4 
[38]. The table shows Accuracy and Recall of Stacking and Voting for full feature 
set and reduced feature sets using CSE and PCA. For more details on CSE and 
PCA, please refer to Sect.4.5. Ensemble learning was used with Naive Bayes, 
Bayesian Networks, Decision trees as base classifiers, and Random forest as the 
meta classifier. 


Our experiments show that Stacking performs better than Voting as number of 
base classifiers increase. However, Stacking suffers from a drawback that it needs 
more ‘build time’ than Voting. 


4.3 Conversation-Based Mechanisms 


Five-tuple flow-based categorization has been the de-facto standard for Internet 
traffic analysis. This classical definition relies on port and protocol. Flow-based 
approaches have seen wide use in problems of Internet traffic classification [21, 22]. 
However, recent literature [26, 33] has reported that modern malware and botnets are 
known to randomize their port numbers and switch between TCP and UDP for their 
communication. Such behavior can allow malware to circumvent traditional security 
mechanisms which rely on ‘flow’ definition. By randomizing ports and switching 
between TCP and UDP, a malicious application creates multiples flows of what is 
actually a single ‘conversation’ happening between a pair of nodes. Even many P2P 
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applications like BitTorrent are known to randomize ports in order to circumvent 
Firewalls which try to block the standard ports used by these applications. 

In this section, we discuss a conversation-based approach targeted toward identifi- 
cation of stealthy botnets in Internet traffic [33]. Stealthy botnets try to evade standard 
detection mechanisms by generating little traffic and perform malicious activities in 
a manner not observable to a network administrator. This may include the Command 
& Control (C & C) activities of a botnet or other nefarious activities like spamming, 
Bitcoin mining, etc. 

A conversation-based approach is port- and protocol-oblivious. Herein, we begin 
by extracting five-tuple flows occurring between all pair of nodes seen in network 
traffic based on a time-out parameter (TIMEGAP). For every pair of nodes, the flows 
are further aggregated in conversations. Multiple flows between two nodes are aggre- 
gated into a single conversation given that the last packet of the previous flow and 
first packet of the next flow occur between a FLOWGAP time period. The FLOWGAP is 
tunable parameter which is to be decided by a network administrator. Certain botnets 
may try to pass under the radars of Intrusion Detection Systems/Firewalls by generat- 
ing very little traffic. A high FLOWGAP can be useful for observing the conversations 
of such stealthy botnets. Our experiments in [33] indicated that a FLOWGAP value 
of one hour is suitable for most purposes. 

It has been reported that a observing a flow for a larger time window of, say, one 
hour aids in higher accuracy of malicious traffic detection [36]. However, a large 
time window will create network traces of huge size. With the increasing size of 
large-scale networks, parsing such traces create a huge performance overhead. A 
conversation-based approach aids in this aspect by following a two-tier architecture. 
In the first step, flows are created from small time windows of, say, ten minutes. 
Parsing flows of ten minute duration does not create a performance overhead. As a 
second step, the flows between two nodes are aggregated into a single conversation 
(based on the FLOWGAP parameter explained above). This approach can be further 
enhanced by clustering the flows created in the first step, and then creating conver- 
sations of flows within each cluster. For more details of this approach, we refer the 
reader to [31]. 

In order to evade detection by Intrusion Detection Systems, malware and bot- 
infected machines often exhibit a ‘low-lying’ behavior. They are stealthy in their 
activities and do not generate much traffic. However, in order to not lose connectiv- 
ity with their peers, bots engage in regular exchange of messages with each other. For 
example, we observed in the network traces of the Zeus botnet that the bot regularly 
sent out as few as one or two packets every half an hour over UDP. Conversation- 
based approaches harness this ‘low volume, high duration’ behavior of stealthy bot- 
nets. Since conversation-based approaches do not rely on port numbers or Transport 
layer protocol for the detection of malicious traffic, they cannot be circumvented by 
bots using these mechanisms. 

After creating conversations between a pair of nodes, statistical features are 
extracted from each conversation. These features are aimed at capturing the ‘low 
volume, high duration’ behavior of stealthy botnets. The features extracted are: 
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1. The duration of the conversation. 

2. The number of packets exchanged in the conversation. 

3. The volume of data exchanged in the conversation. 

4. The median value of the inter-arrival time of packets in that conversation. 


These features are used to build classification modules utilizing supervised Machine 
learning algorithms for separating malicious traffic from benign traffic. We employed 
three supervised machine learning algorithms—Bayesian Network, J48 Decision 
trees, and Boosted Reduced error-pruning (REP) trees. The classification results 
(adopted from [33]) obtained with the data of two malicious applications (namely 
Storm and Waledac) and two P2P applications (namely uTorrent and eMule) are 
given in Fig. 10. TP rate represents “True Positive rate,’ and FP rate represents ‘False 
Positive rate.’ An outline of this approach is given in Fig. 9. 


Network Traffic 





Benign P2P P2P botnets 


Packet Filtering module 





parse packets discard invalid packets 


4 


Flow creation module 





5-tuple flows based on TIMEGAP 


Conversation generation module 





2-tuple conversations based on FLOWGAP 


Classification module 


| 


Bayesian Network J48 Decision Trees Boosted REP trees 


Fig.9 System overview: conversation-based mechanisms 
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Fig. 10 Botnet Traffic Identification with Conversation-based Mechanisms 


4.4 Signal-Processing Approaches with Machine Learning 


In this section, we discuss a signal-processing-based approach toward identification 
of malicious traffic [32]. By utilizing signal-processing approaches, we aim at iden- 
tifying malicious botnet traffic by focusing on their timing and data patterns. The 
traffic of botnets frequently exhibits regularities in timing and payload sizes [47]. We 
leverage this information for the detection of infected hosts. Internet traffic between 
a pair of nodes seen in network communication is modeled as a ‘signal.’ For each 
signal, several features based on Fourier Transforms (DFTs) and Shannon’s Entropy 
theory are extracted. These features are explained below: 

Fourier Transform-based features: Given a vector of n input amplitudes, the 
Discrete Fourier Transform yields a set of n frequency magnitudes: 





lees jzkn 
X[k] = X x(n) -e (1) 
n=0 


Here, N is the length of the sequence to be transformed, k is used to denote the 
frequency domain ordinal, and n is used to represent the time-domain ordinal. 

We apply Fourier Transform to the Inter-arrival time of packets in a conversa- 
tion and to the payload length (in Kbs) per packet in a conversation. In any time 
series, most of the energy is contained in the first few Fourier Transform coefficients 
itself. Thus we sort the Fourier Transform values by magnitude, and retain the top 
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10 Fourier Transform magnitude values and corresponding phase values. We also 
retain the magnitude and phase of the ‘prime wave’ obtained by the vector sum of 
the top ten peaks. 

Shannon’s Entropy-based feature: Botnets primarily engage in exchange of 
C & C information. Their communication frequently occurs in small, fixed-size pay- 
loads. On the other hand, Internet traffic exhibits much more variation in payload 
sizes. This can be attributed to the fact that different users of the Internet use differ- 
ent applications. The combination of multiple users and multiple applications leads 
to greater variety. 

We quantify the amount of randomness or entropy in payload variation in each 
conversation in the form of “Compression ratio’. We model payload size per packet as 
a discrete symbol, and compute the expected compression ratio based on Shannon’s 
Entropy theory. The Entropy H(X) is given by: 


H(X)= -9 p(x) log) (2) 


xEeX 


Benign conversations are expected to have high entropy, which will lead to less 
compression. On the other hand, botnet conversations are expected to have low 
entropy, leading to higher compression rates. 

We combine our signal-processing-based features with traditional network 
behavior-based features for each conversation (such as, Average payload size, Num- 
ber of packets, Median of inter-arrival time between packets, etc.). These set of fea- 
tures were used to build detection models with supervised Machine learning algo- 
rithms. For more details, we refer the reader to [32]. 


4.5 Feature Subset Selection Approaches 


Feature subset selection approaches can be utilized to select a subset of features 
which highly contribute towards the classification, and remove those features which 
do not provide high contribution toward the classification problem. The number of 
features used in training Machine learning models has a direct relationship with the 
‘build time’ of the model. By getting rid of irrelevant features, computational com- 
plexity of training the models as well as classification can be reduced. 

We measure the impact of feature subset selection on three well-known Machine 
learning algorithms—Naive Bayes, Bayesian Network and J48 Decision trees. We 
used three feature subset selection approaches—Correlation-based Feature Selection 
(CFS) [16], Consistency-based Subset evaluation (CSE) [7] and Principal Compo- 
nent Analysis (PCA) [20]. We compared the build times of each of these approaches 
with the build time for the full feature set [34]. We observed a pattern in build times 
for different models, irrespective of the classifier used. Higher number of features 
always led to more build time, while lesser number of features always assured lesser 
building time for the model. Thus, the full feature set had the maximum build time. 
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The build time of CFS, CSE, and PCA is in the order of the number of features 
selected by them. However, we clearly observed that Decision trees, in general, take 
long time to build. Naïve Bayes classifier was the fastest, followed by Bayesian net- 
work. 


4.6 Graph Partitioning Approaches for Botnet Traffic 
Identification 


Conventional graph partitioning algorithms—such as community detection algo- 
rithms, sybil detection algorithms, etc.—leverage the presence of a bottleneck ‘cut’ 
separating a subgraph of interest from the rest of the graph. These approaches are 
designed for vertex partitioning. This is not applicable for identification of botnets in 
network traffic [30]. Botnet edges and edges corresponding to legitimate traffic can- 
not be separated by a bottleneck cut since they can belong to the same node. Thus, 
identification of botnet traffic is better defined as an edge-partitioning problem from 
graph-theoretic perspective. 

Most of the previous botnet detection approaches leveraging graph theory rely on 
vertex partitioning approaches. BotTrack [14] attempts to identify highly connected 
subgraphs of the traffic dispersion graph. It uses the page-rank algorithm to obtain 
page-rank scores corresponding to ‘hub’ and ‘authority’ values. The nodes in the 
graph are then clustered based on the two dimensional vector of hub and authority 
value. A well-known community detection algorithm, the Louvain algorithm [3], has 
been used by Ruehrup et al. [41] and Hang et al. [17] to detect ‘botnet communities’ 
in network traffic. Since these approaches rely on vertex-partitioning techniques, they 
will be unable to categorize whether a node belongs to two (or more) communities 
(say, one ‘botnet’ community, and another benign community). 

Botyacc [30] is a recent work which uses edge-partitioning approach for P2P bot- 
net detection in large networks. However, neither Botyacc nor most of past research 
employing graph theory has evaluated the detection of P2P botnets in the presence 
of benign P2P traffic. Detection of P2P botnets in the presence of benign P2P appli- 
cations such as uTorrent, eMule, etc. is a challenging task since both these traffic 
utilize P2P communication topologies. Thus, if a system running a P2P application 
gets infected by a P2P botnet, such approaches may not be able to correctly catego- 
rize its traffic. 

We evaluate vertex-partitioning and edge-partitioning approaches for the detec- 
tion of P2P botnets in the presence of benign P2P traffic in a network. For vertex 
partitioning, we resort to the well-known approach of Louvain algorithm. For edge- 
partitioning, we use a recent approach of ‘link communities’ given by Ahn et al. 
[1]. We created a graph from network flows of one P2P application eMule and two 
P2P botnets, Zeus, and Waledac. The data of eMule had one source node engaged 
in P2P communication with several destination nodes. The data of Zeus contained 
one source node, and that of Waledac contained three source nodes. We overlapped 
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the eMule node with one of the Waledac nodes. The graph thus obtained contains a 
P2P botnet and a P2P application overlapped on a single node. We then used Lou- 
vain algorithm for vertex-partitioning and Link communities algorithm for edge- 
partitioning. The resultant graphs are given in Fig. 11. 

As expected, we see Zeus as a separate community in both the graphs (repre- 
sented in red in Fig. lla and in green in Fig. 11b). Louvain algorithm could create 
two communities for the Waledac-eMule subgraph. In reality, however, this should 
have been four communities—three for the three source nodes of Waledac, and one 
for the one source node of eMule. 





Fig. 11 Vertex-partitioning and Edge-partitioning approaches with P2P botnets and P2P applica- 
tions 


% of malicious nodes 


Fig. 12 Sample results obtained using edge-partitioning approach 
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With the edge-partitioning approach, we obtained a total of 56 communities. Out 
of these, only four communities were prominent, and others contained only three or 
four nodes. We have shown only these prominent communities in the Fig. 11b. This 
approach was successful in creating four overlapping communities for the Waledac- 
eMule subgraph. 

We calculate the percentage of malicious edges in different communities attached 
to a particular node, and then obtain a weighted average of that percentage. In this 
way we arrive at the maliciousness of a particular node. Sample results obtained 
from this approach are given in Fig. 12, wherein the percentage of malicious nodes 
detected in a community are indicated. 

These edge-partitioning approaches, complemented with techniques such as Boty- 
acc, can aid in correct identification of nodes which are running P2P applications and 
are infected by P2P botnets at the same time. 


5 Conclusion 


This chapter presented a case study of identification of unwanted traffic at a large- 
scale University network. Initially, we discussed about traditional approaches for 
detection of unwanted traffic. However, with the advancement of protocols and intro- 
duction of newer applications every year, most of these techniques have become 
obsolete. 

Most of the critical network management activities are performed by automated 
network appliances. However, these devices come with multiple functions 
that degrade their efficacy. In this paper, we have brought out techniques based on 
Machine learning approaches to clearly segregate malicious traffic from 
non-malicious traffic without performing Deep Packet Inspection that would oth- 
erwise have been costly. The intelligent approaches proposed in this work clearly 
separate out P2P from non-P2P traffic and also between benign and malicious P2P 
traffic. The approaches proposed have been tested on a realistic University network 
that consists of different types of data that come inside the network at unprecedented 
scale and speed. We plan to extend this work by adding more number of features 
selected through an optimal means to further accurately classify the network logs 
and identify malicious traffic within those. 
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Application-Level Benchmarking of Big 
Data Systems 


Chaitanya Baru and Tilmann Rabl 


Abstract The increasing possibilities to collect vast amounts of data—whether in 
science, commerce, social networking, or government—have led to the “big data” 
phenomenon. The amount, rate, and variety of data that are assembled—for almost 
any application domain—are necessitating a reexamination of old technologies and 
development of new technologies to get value from the data, in a timely fashion. 
With increasing adoption and penetration of mobile technologies, and increasing 
ubiquitous use of sensors and small devices in the so-called Internet of Things, the 
big data phenomenon will only create more pressures on data collection and pro- 
cessing for transforming data into knowledge for discovery and action. A vibrant 
industry has been created around the big data phenomena, leading also to an 
energetic research agenda in this area. With the proliferation of big data hardware 
and software solutions in industry and research, there is a pressing need for 
benchmarks that can provide objective evaluations of alternative technologies and 
solution approaches to a given big data problem. This chapter gives an introduction 
to big data benchmarking and presents different proposals and standardization 
efforts. 


1 Introduction 


As described in [11], database system benchmarking has a rich history, from the 
work described in the paper entitled, A Measure of Transaction Processing Power 
[1], to the establishment of the Transaction Processing Council in 1988, and con- 
tinuing into the present. The pressing need for database benchmark standards was 
recognized in the mid to late 1980s, when database technology was relatively new, 
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and a number of companies were competing directly in the database systems 
software marketplace. The initial efforts were simply on persuading competing 
organizations to utilize the same benchmark specification—such as the DebitCredit 
benchmark introduced in [12]. However, the benchmark results were published 
directly by each company, often eliminating key requirements specified in the 
original benchmark. Thus, there was need for a standard-based approach, along 
with a standard organization that could uniformly enforce benchmark rules while 
also certifying results produced, thereby providing a stamp of approval on the 
results. Thus was born the Transaction Processing Performance Council, or TPC, 
in 1988. With its early benchmarks, such as TPC-C, TPC-D, and TPC-H, the TPC 
was successful in producing widely used benchmark standards that have led directly 
to database system product improvements and made a real impact on product 
features. 

With the rapid growth in big data’ applications, and vendor claims of hardware 
and software solutions aimed at this market, there is once again a need for objective 
benchmarks for systems that support big data applications. In the emerging world of 
“big data,” organizations were once again publishing private benchmark results, 
which claimed performance results that were not audited or verified. To address this 
need, a group from academia and industry (including the authors) organized the first 
Workshop on Big Data Benchmarking (WBDB) in May 2012, in San Jose, Cali- 
fornia. Discussions at the WBDB workshop covered the full range of issues, and 
reinforced the need for benchmark standards for big data. However, there was also 
recognition of the challenge in defining a commonly agreed upon set of big data 
application scenarios that could lead toward benchmark standards. 

In retrospect, the early TPC benchmarks had an easier time in this regard. Their 
initial focus was transaction processing—typically defined by insert, update, delete, 
and read operations on records or fields within records. Examples are point-of-sale 
terminals in retail shopping applications, bank teller systems, or ticket reservation 
systems. Subsequent benchmarks extended to SQL query processing with relational 
database systems. Big data applications scenarios are, however, much more varied 
than transaction processing plus query processing. They may involve complex 
transformation of data, graph traversals, data mining, machine learning, sequence 
analysis, time series processing, and spatiotemporal analysis, in addition to query 
processing. The first challenge, therefore, is to identify application scenarios that 
capture the key aspects of big data applications. Application-level data benchmarks 
are end-to-end benchmarks that strive to cover the performance of all aspects of the 
application, from data ingestion to analysis. 

A benchmark specification must pay attention to multiple aspects, including 


(a) The so-called system under test (SUT), i.e., the system or components that are 
the focus of the testing, which may range from a single hardware or software 
component to a complete hardware or software systems. 


'The term “big data” is often written with capitals, i.e. Big Data. In this paper, we have chosen to 
write this term without capitalization. 
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(b) The types of workloads, from application-level to specific component-level 
operations. Component benchmarks focus on specific operations; examples are 
I/O system benchmarks, graphics hardware benchmarks, or sorting bench- 
marks. The types of workloads range from very simple micro-benchmarks, 
which test a certain type of operation, to complex application simulations or 
replay of real workloads. 

(c) The benchmarking process. Kit-based benchmarks provide an implementation 
or suite of tools that automates the benchmarking process. Specification-based 
benchmarks describe the detailed benchmarking process and allow for dif- 
ferent implementations of the benchmark. The former are typically used in 
component benchmarks, while the latter are used for database, end-to-end 
benchmarks. 

(d) The target audience. Benchmark details and especially the representation of 
the results may differ depending upon the target audience for the benchmark. 
End users and product marketing may require results that are easily compa- 
rable with realistic workloads. Performance engineers may prefer workloads 
that cover typical modes of operation. System testers may want to cover all 
modes of operation, but also need deep insights into the system behavior. Big 
data benchmarks exist in all of the forms described above. 


While benchmarks can only represent real-world scenarios—and are not the 
real-world scenarios themselves—they nonetheless play an essential role. They can 
represent a broad class of application needs, requirements, and characteristics; 
provide repeatability of results; facilitate comparability among different systems; 
and provide efficient implementations. A good benchmark would represent the 
important aspects of real-world application scenarios as closely as possible, provide 
repeatability and comparability of results, and would be easy to execute. 

The rest of this paper is structured as follows. Section 2 provides examples of 
some big data application scenarios. Section 3 describes useful benchmark 
abstractions that represent large classes of big data applications. Section 4 describes 
the approach taken by different benchmark standards, such as TPC and SPEC. 
Section 5 describes current benchmarking efforts, and Sect. 6 provides a conclusion. 


2 Big Data Application Examples 


While there is broad consensus on the potential for big data to provide new insights 
in scientific and business applications, characterizing the particular nature of big 
data and big data applications is a challenging task—due to the breadth of possible 
applications. In 2014, the US National Institute for Standards and Technologies 
(NIST) initiated a NIST Big Data Public Working Group (NBD-PWG) in order to 
tackle the issue of developing a common framework and terminology for big data 
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and big data applications.” As documented in the NBD-PWG volume on Use Cases 
and General Requirements,’ the range of real-world big data applications is broad, 
and includes collection and archiving of data; use in trading and financial sector; 
delivery of streaming content in a variety of applications scenarios including, for 
example, security, entertainment, and scientific applications; indexing to support 
web search; tracking data streams related to shipping and delivery of physical items 
(e.g., by FedEX, UPS, or US Postal Service); collection and analysis of data from 
sensors, in general; personalized health, precision medicine and other applications 
in healthcare and life sciences (including electronic medical records, pathology, 
bioimaging, genomics, epidemiology, people activity models, and biodiversity); 
deep learning with social media and a variety of other data; processing for driverless 
cars; language translation; smart grids; and others. 

One of the example applications in the list of NIST Uses Cases is the Genome in 
a Bottle Consortium*—a public—private—academic consortium hosted by NIST to 
develop the technical infrastructure needed to enable translation of whole human 
genome sequencing to clinical practice. A current pilot application at NIST, 
employing open-source bioinformatics sequencing software on a 72-core cluster, 
has generated 40 TB of data. However, DNA sequencers will be able to gener- 
ate ~300 GB compressed data per day in the near future. An individual lab will 
easily be able to produce petabytes of genomics data in future. 

In another example, the Census Bureau is considering the use of advanced 
techniques to conduct demographic surveys, whose costs are increasing even as 
survey responses decline. These techniques include recommendation systems to 
help improve response rates; use of “mash ups” of data from multiple sources; and 
the use of historical survey administrative data, to help increase the overall survey 
data quality while simultaneously reducing their costs. As described in [ref], the US 
Census is planning to use big data in multiple ways including for “Address can- 
vassing,”’ i.e., to validate, correct, or delete existing Census Bureau addresses; to 
assist in the planning of the survey itself; and to assist in nonresponse follow-up. 

A third use case deals with large-scale deep learning models. Deep learning 
algorithms employ architectures with multiple levels of nonlinear transformations, 
where the output of each layer is provided as input to the next. The demonstrated 
ability of deep learning models to extract representations and information from big 
data have made them attractive approaches for a number of big data applications 
including, for example, image and speech recognition. With increasingly sophis- 
ticated speech-based interfaces to computers and devices, and the need for image 
processing/recognition in applications like self-driving cars, the big data needs will 
only increase in this area. 

While these examples provide a glimpse of the vast potential of data-intensive 
approaches in real applications, they also illustrate the challenges in defining the 


°NIST Big Data Public Working Group, http://bigdatawg.nist.gov/home.php. 
NIST NBD-PWG Use Cases and Requirement, http://bigdatawg.nist.gov/usecases.php. 
*Genome in a Bottle Consortium, https://sites.stanford.edu/abms/giab. 
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scope of the benchmarking problem for big data applications. The next section 
provides two specific approaches to tackling this issue. 


3 Levels of Abstraction 


Since the scope of big data applications can be vast—as described in the previous 
section—it is important to develop “abstractions” of real applications, in order to 
then develop benchmark specifications, which are based on those abstractions. The 
two abstractions described in this section are based on (1) extending the familiar 
data warehouse model to include certain big data characteristic in the data and the 
workload and (2) specifying a pipeline of processing, where data is transformed and 
processed in several steps. The specifics of each step may be different for different 
applications domains. 

As described in Sect. 1, TPC benchmarks provide an application-level abstrac- 
tion of business applications, such as transaction processing and/or complex query 
processing. TPC benchmarks, such as TPC-C, TPC-H, TPC-DS,° model retail 
transaction processing and data warehousing environments. The database model and 
workload provide a representative view of a specific business application scenario. 
Nonetheless, the results of such benchmarks can be used as a guide for a variety of 
other application (non-business) use cases with similar characteristics. Following 
this example, we present two models for big data benchmarks, in this section. The 
first, BigBench, follows the TPC model; while the second, Deep Analytics Pipeline, 
is a model based on a generalization of big data processing pipelines. 


3.1 BigBench 


BigBench is a big data analytics benchmark based on TPC-DS. Its development was 
initiated at the first Workshop on Big Data Benchmarking in May 2012 [6]. BigBench 
models a retail warehouse that has two sale channels: web sales and store sales. An 
excerpt of the data model is shown in Fig. 1. In order to appropriately reflect the big 
data use case, BigBench features not only structured data but also semi-structured and 
unstructured data. These data sections contain dependencies. For example, the web 
log, or clickstream data (semi-structured), has references to the SALES table. There 
are thirty queries specified in the BigBench workload, which cover a broad variety of 
analytics representing the different big data levers that were identified by Manyika 
et al. [8]. The queries use cases cover business aspects of marketing, merchandising, 
operations, supply chains, and reporting in a typical enterprise. 

The BigBench processing model is targeted at batch analytics. The complete 
benchmark process consists of three stages: data generation, data load, a Power 


>TPC, http://www.tpc.org/. 
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Fig. 1 BigBench data model 


Test, and a Throughput Test. The data generation step generates the complete data 
set in different flat file formats. However, it is not part of the measured benchmark 
run. In the loading stage, data can be loaded into the system under test. The Power 
Test is performed by a serial execution of all thirty queries in the workload, while 
the Throughput Test consists of running a preselected number of serial streams, 
each of which is a permutation of the thirty queries. The total number of queries run 
is divided by the benchmark runtime to obtain a queries-per-hour metric. 

BigBench has gained widespread attention in industry and academia and is 
currently in process to be standardized by the TPC. Several extensions were pro- 
posed [5]. 


3.2 Data Analytics Pipeline 


Another proposal for a big data benchmark, referred to as the Deep Analytics 
Pipeline, is based on the observation that many big data applications are required to 
process streams of data, which requires a pipelined approach to data processing [4]. 
Figure 2, from [4], shows the steps in such a pipeline. This pipelined approach to 
processing big data is, indeed, a generic processing “pattern” that applies to a wide 
range of application scenarios. For example, processing of user clickstream data to 
predict ad placement in online advertising applications; predicting customer churn 
in, say, the banking industry based on user interactions as well as demographics, 
income, and other customer data; predicting insurance fraud based on a customer’s 
activities; and predicting a patient’s likelihood to visit the emergency room, based 
again on their activity and health information. All these examples involve using a 
variety of data related to the entity of interest, e.g., customer, patient, online user, 
etc., and detecting correlations between “interesting” outcomes and prior behavior. 
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Fig. 2 Deep analytics pipeline 


In essence, all such applications are performing a “user modeling” exercise based 
on transactional data and other historical and related information. 

Pipelined data processing is also very typical of many, if not most, scientific 
applications as well. The processing pipeline in this case includes steps of data 
acquisition, data ingestion (data cleaning and normalization), data validation, and a 
variety of downstream analytics and visualization. The pipeline model is, thus, 
generic in nature. Different classes of application may be characterized by varia- 
tions in the steps of the pipeline. 

For benchmarking purposes, the pipeline stages are described based on their 
function, rather than in platform-specific terms. The pipeline stages could be 
implemented on a single hardware platform, or on distributed platforms. 


4 Benchmarks Standards 


Current industry standards provide two successful models for data-related bench- 
marks, TPC and SPEC, both of which were formed in 1988. TPC was launched 
with the objectives of creating standard benchmarks and a standard process for 
reviewing and monitoring those benchmarks [11]. SPEC was founded in the same 
year by a small number of workstation vendors who realized that the marketplace 
was in desperate need of realistic, standardized performance tests. The key real- 
ization was that an “ounce of honest data was worth more than a pound of mar- 
keting hype.” Interestingly, the current community interest in big data 
benchmarking has the same motivation! Both organizations, TPC and SPEC, 
Operate on a membership model. Industry as well as academic groups may join 
these organizations. 


4.1 The TPC Model 


TPC benchmarks are free for download; utilize standardized metrics for measuring 
transaction and query throughput; measure performance as well as price perfor- 
mance of given solutions; and have more recently introduced an energy metric, to 
measure performance versus energy consumption. 
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TPC benchmarks are designed to test the performance of the entire system— 
hardware as well as software, using metrics for transactions or queries per unit of 
time. The specifications are independent of the underlying hardware and software 
implementations. Over its history, TPC has demonstrated that its benchmarks have 
relatively long “shelf life.” Benchmarks remain valid for several years and, in the 
case of TPC-C, it is worth noting that the 22-year-old benchmark is still valid. The 
benchmark measures transactions per minute for a scenario based on order-entry 
systems. The transactions include entering and delivering orders, recording pay- 
ments, checking the status of orders, and monitoring the level of stock at ware- 
houses. One of the keys to the longevity of the TPC-C benchmark is the rule for 
“data scaling,” which is based on a “continuous scaling” model, where the number 
of warehouses in the database scales up with the number of transactions. 

TPC benchmark “sponsors” may publish official TPC results for a fee. The 
publication rules require full disclosure of all information, including system con- 
figuration, pricing, and details of performance. Benchmark results are audited by an 
independent, third-party auditor. 

TPC benchmarks that are query processing-oriented specify fixed database sizes 
at which the benchmark may be executed. These so-called database scale factors 
range from 1—representing a 1 GB “raw” database size—to 100,000 for a 100 TB 
raw database size. The most common scale factors for which benchmarks have been 
published are ranging from 100 to 10,000 (100 GB to 10 TB). Big data benchmarks 
may also need to adopt a similar scheme for database sizes. 


4.2 The SPEC Model 


SPEC benchmarks typically focus on specific functions or operations within a 
system, e.g., integer performance, sort performance. The benchmarks are typically 
server-centric, and test performance of small systems or components of systems. 
Unlike TPC, each benchmark defines its own metric, since different benchmarks 
may focus on different aspects of a system. As a result, they tend to have short shelf 
life—benchmarks can become obsolete with a new generation of hardware or 
system. The SPEC benchmark toolkits can be downloaded for a fee. 

Publication of benchmark results by SPEC is free to members and subject to a 
modest fee for nonmembers. Unlike TPC, which incorporates third-party audits, 
SPEC benchmark results are peer reviewed. Also, unlike TPC, which requires 
extensive, full disclosure, SPEC requires only a disclosure summary—partly because 
the benchmark is at a component level within a system. The following table—from a 
tutorial on Big Data Benchmarking presented by Baru and Rabl at the IEEE Big Data 
Conference, 2014°—summarizes the features of TPC versus SPEC benchmarks. 





http ://www.slideshare.net/tilmann_rabl/ieee2014-tutorialbarurabl?qid=d2 1f7949-d467-4824- 
adb 1-2e394e9a2239&v=&b=&from_search=1. 
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TPC model SPEC model 

Specification based Kit based 

Performance, price, energy in one Performance and energy in separate 
benchmark benchmarks 

End-to-end benchmark Server-centric benchmark 

Multiple tests (ACID, load, etc.) Single test 

Independent review Peer review 

Full disclosure Summary disclosure 


4.3 Elasticity 


An important requirement for big data systems is elasticity. For big data systems 
that deal with large data that are continuously growing, e.g., clicks streams and 
sensor streams, the system must be designed to automatically take advantage of 
additional resources, e.g., disks or nodes, as they are added to the system. Con- 
versely, given the scale of many big data systems, there is a high probability of 
component failures during any reasonable workload run. The loss of a component 
in such a system should not lead to application failure, system shutdown, or other 
catastrophic results. The system should be “elastic” in how it also adopts to addition 
and/or loss of resources. Benchmarks designed for big data system should attempt 
to incorporate these features as part of the benchmark itself, since they occur as a 
matter of course in such systems. Currently, TPC benchmarks do require ACID 
tests (for testing atomicity, consistency, isolation, and durability properties of 
transaction processing and/or database systems). However, such tests are done 
outside the benchmark window, 1.e., they are not a part of the benchmark run itself, 
but are performed separately. 


5 Current Benchmarking Efforts 


Several efforts have been created to foster the development of big data benchmarks. 
A recent paper summarizes a number of such efforts [7]. The first Workshop on Big 
Data Benchmarking, which was held in San Jose, California, USA, in May 2012, 
created one of the first community forums on this topic. In this workshop, 60 
participants from industry and academia came together to discuss the development 
of industry standard big data benchmarks. The workshop resulted in two publica- 
tions [2, 3] and the creation of the big data Benchmarking Community (BDBC), an 
open discussion group that met in biweekly conference calls and via online 
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discussions. In mid-2014, with the creation of the SPEC Research Group on big 
data,’ the BDBC group was merged with the SPEC activity. With the formation of 
the SPEC RG, weekly calls have been established, as part of the SPEC activity, 
with the weekly presentations alternating between open and internal calls. The open 
presentations cover new big data systems, benchmarking efforts, use cases, and 
related research. The internal calls are restricted to SPEC members only, and focus 
on discussion of big data benchmark standardization activities. 

The WBDB series launched in 2012 has been continuing successfully, with 
workshops in the India (December 2012), China (July 2013), US (October 2013), 
Germany (October 2014), and Canada (June 2015). The next workshop, the 7th 
WBDB, will be held on December 14-15 in New Delhi, India. The WBDB 
workshops have, from the beginning, included participation by members of stan- 
dards bodies, such as SPEC and TPC. Workshop discussions and papers presented 
at WBDB have led to the creation of the SPEC Research Group on big data, as 
mentioned earlier, and creation of the TPC Express Benchmark for Hadoop Sys- 
tems (aka TPCx-HS), which is the first industry standard benchmark for Apache 
Hadoop compatible big data systems, and is based on the Terasort benchmark. The 
BigBench paper presented first at WBDB has also led to the formation of the 
TPC—BigBench subcommittee, which is working toward a big data benchmark 
based on a data warehouse-style workload. 

Finally, an idea that has been discussed at WBDB is the notion of creating a 
BigData Top100 List®, based on the well-known TOP500 list used for supercom- 
puter systems. Similar to the TOP500, the BigData Top100 would rank the world’s 
fastest big data systems—with an important caveat. Unlike the TOP500 list, the 
BigData Top100 List would include a price/performance metric. 


6 Conclusion 


Big data benchmarking has become an important topic of discussion, since the 
launching of the WBDB workshop series in May 2012. A variety of projects in 
academia as well as industry are working on this issue. The WBDB workshops 
have provided a forum for discussing the variety and complexity of big data 
benchmarking—including discussions of who should define the benchmarks, e.g., 
technology vendors versus technology users/customers; what new features should 
be include in such benchmarks, which have not been considered in previous per- 
formance benchmarks, e.g., elasticity and fault tolerance. 

Even though this is a challenging topic, the strong community interest in 
developing standards in this area has resulted in the creation of the TPCx-HS 
benchmark; formation of the TPC-BigBench subcommittee; and the formation of 


SPEC RG big data - http://research.spec.org/working-groups/big-data-working-group. 
‘BigData Top100 - http://www.bigdatatop100.org/. 
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the SPEC research group on big data. Finally, a benchmark is only a formal, 
standardized representation of “typical” real-world workloads that allows for 
comparability among different systems. Eventually, users are interested in the 
performance of their specific workload(s) on a given system. If a given workload 
can be formally characterized, it could then be executed as a service across many 
different systems, to measure the performance of any system on that workload. 
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Managing Large-Scale Standardized 
Electronic Health Records 


Shivani Batra and Shelly Sachdeva 


Abstract Electronic health records (EHRs) contain data about a person’s health 
history. Increasingly, EHRs have the characteristics of big data in terms of their 
volume, velocity, and variety (the 3 “V’’s). Volume is a major concern for EHRs 
especially due to the presence of huge amount of null data, i.e., for storing sparse 
data that leads to storage wastage. Reducing storage wastage due to sparse values 
requires amendments to the storage mechanism that stores only non-null data, and 
also allows faster data retrieval and supports multidimensional heterogeneous data. 
Another area of concern regarding EHRs data is standardization. Standardization 
can aid in semantic interoperability that resolves the discrepancies in interpretation 
of health records among different medical organizations or persons involved. 
Various proposals have been made at the logical layer of relational database 
management system for managing large-scale standardized records in terms of data 
volume, velocity, and variety. Every proposed modification to logical layer has its 
pros and cons. In this chapter, we will discuss various aspects of the solutions 
proposed for managing standardized EHRs, and the approaches to adopt these 
standards. After efficient management of EHR data, analytics can be applied to 
minimize the overall cost of healthcare. 
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1 Introduction 


In today’s world of digitization, a large amount of the generated data is stored 
electronically. Healthcare is a major domain with lots of data. For better utilization 
of health records, they are stored electronically. Not surprisingly, electronic health 
records (EHRs) data have a multitude of representations [1]. EHRs contain longi- 
tudinal (life-long) health-related information of a person. The contents may be 
structured, semi-structured, or unstructured, or a mixture of all these. In fact, these 
may be plain text, coded text, paragraphs, measured quantities (with values and 
units); date, time, date—time (and partial date/time); encapsulated data (multimedia, 
parsable content); basic types (such as, Boolean, state variable); container types 
(list, set); and uniform resource identifiers (URI). Scope of primary care is limited 
to one hospital, whereas an integrated care constitutes multiple hospitals. An 
integrated care EHR [2] is defined as “a repository of information regarding the 
health of a subject of care in computer processable form, stored and transmitted 
securely, and accessible by multiple authorized users. It has a commonly agreed 
logical information model which is independent of EHR systems. Its primary 
purpose is the support of continuing, efficient and quality integrated healthcare and 
it contains information which is retrospective (a historical view of health status and 
interventions), concurrent (a “now” view of health status and active interventions) 
and prospective (a future view of planned activities and interventions).” 

Modern medicine is a data-intensive practice that generates large volumes of 
complex and heterogeneous data. Figure 1 shows the complexity of the healthcare 
domain. It is vast, consisting of patients, clinical concepts, relationships, termi- 
nology concepts, and terminology relationships. There are a large number of 
patients and clinical concepts. A clinical concept is basically a specific disease or 
fact such as, blood pressure or heart rate. One clinical concept might have some 
relationship (shown as EHR_relationship in Fig. 1) with other clinical concepts. For 
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Fig. 1 Complexity of the Healthcare domain 
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example, increased heart rate associated with increased blood pressure may result in 
hypertension or cardiovascular disease. 

The use of medical terminologies constitutes an important mechanism to stan- 
dardize nomenclature and to introduce semantics to EHRs [1]. Clinicians often use 
strange terminology and a variety of codes to describe diagnosis. Also, physicians 
have different specialties. There are many standardized coding systems for medical 
terminology. Each defines its own set of terms and relation among terms. For 
example, the Systemized Nomenclature of Medicine—Clinical Terms (SNOMED 
CT) [3] coding system alone defines more than 400,000 clinical concepts, 1 million 
terms and 1.6 million relationships. A patient has one-to-many relation with clinical 
concepts. A clinical concept has one-to-many relation with EHR relationships. 
A terminology concept has one-to-many relation with clinical concepts. As 
healthcare data is becoming more complex, there is a need to handle such com- 
plexity. The challenges include capturing, storing, searching, sharing, and analyzing 
EHR data. 

With every passing year, the healthcare costs keep rising. As per national health 
expenditure data maintained by the United States Centers of Medicare & Medicaid 
Services [4, 5], in 1963, US healthcare expenditure was just $35 billion that 
increased to $103 billion in 1973 with an average annual growth rate of 11.5 %. In 
year 2013, US health expenditure was approximately $3 trillion and it is estimated 
to be $5.1 trillion in year 2023, i.e., an average annual growth expected during the 
decade 2013-2023 is 5.8 %. One of the major reasons behind the increase in cost of 
healthcare is the exponential growth of EHRs data. 


1.1 Electronic Health Records Growing BIG 


While electronic medical records (EMRs) maintain information regarding a 
patient’s medical records from a particular practice, EHRs capture longitudinal 
information related to an individual’s overall health. EMRs are intended primarily 
for local use and hence lack in terms of providing semantic interoperability. Indeed, 
absence of support for semantic interoperability can hinder worldwide adoption of 
any healthcare information system. 

EHRs offer a more general digital solution to the world of healthcare that 
otherwise runs on a chain of paper files. There has been a tremendous increase in 
the daily generation of EHR data. In an usual hospital scenario, a patient’s vitals 
such as, temperature, heart beat, and blood pressure are recorded multiple times 
every day. Recording each and every detail of a patient leads to huge volume of 
medical data. With the exponential growth of EHRs, the proverbial 3 V’s (Volume, 
Velocity, and Variety) of big data are becoming more relevant. Here, volume 
signifies the size of EHR data. Velocity refers to the access speed expected from a 
typical clinical information system, and variety to the diverse types of data struc- 
tures involved in EHRs. 
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In 2012, worldwide digital healthcare data was estimated to be equal to 500 
petabytes and is expected to reach 25,000 petabytes in 2020 [6]. It is expected that 
the size of health data will soon increase to zettabytes (equivalent 10°' gigabytes) 
and even yottabytes (10°* gigabytes) [7]. 

Considering the challenges of the healthcare domain, however, big data is not 
just about size. It also involves obtaining insights from complex, noisy, heteroge- 
neous, longitudinal, and voluminous data. Goal of EHR is not just limited to 
capturing large amounts of data, but to improve patient care, patient heath, and to 
decrease overall cost. Trading, auditing, analyzing, and displaying of results are 
made possible by EHR data management and analytics. 

Analyzing big data to obtain new insights is routinely conducted by pharma- 
ceutical industry experts, providers, doctors, insurance experts, researchers, and 
customers. For instance, researchers can mine adverse health events data to detect 
the severity of effects upon taking certain combinations of drugs. Similarly, 
healthcare analysts and practitioners can efficiently handle and make inferences 
from voluminous and heterogeneous healthcare data to improve the health of 
patients while decreasing the costs of care. It has been estimated that big data 
analytics can help in saving approximately $300 billion per year in the US [8]. 
Analyzing EHRs big data can help in reducing waste and inefficiency in clinical 
operations, research and development, public health programs, evidence-based 
medicine, genomic medicine, pre-adjudication fraud analysis, device/remote mon- 
itoring, and predict health events and outcomes [9]. It has been reported that 
analysis of EHRs provided decision support and thus, improved healthcare services 
at approximately 350 hospitals which helped in saving estimated 160000 lives and 
reduced health spending by nearly $13.2 billion in the year 2008 [10]. 

For efficient analytics and cost reduction, the 3Vs of big data must be handled 
efficiently. For managing large volumes of EHR data constituting a variety of data 
types and demanding high access velocity, it is important that the storage issues be 
resolved first. 


1.2 Storage Issues for EHRs 


As EHR data volume increases, the need for efficiently managing the data is also 
becoming prominent. While dealing with EHR data, three issues—Standardization, 
Sparseness, and Volatility—are important from storage point of view. 

Standardization: Standardization of EHRs is the key to support semantic 
interoperability. National Health Information Network (NHIN) of USA defines 
semantic interoperability as “the ability to interpret, and, therefore, to make effec- 
tive use of the information so exchanged” [11]. Similarly, Institute of Electrical and 
Electronics Engineers (IEEE) Standard 1073 defines semantic interoperability as 
shared data types, shared terminologies, and shared codings [12]. Standards provide 
a definitional basis for interoperability [13]. Achieving standardization thus helps in 
global acceptability, interoperability, and higher usability [14]. 
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Sparseness: One of the major concerns regarding the volume of EHRs is the 
presence of large number of null values that leads to sparseness. A typical schema 
for EHRs must constitute each and every attribute contributing to healthcare. 
However, whenever a patient visits a doctor, typically only a few of his/her vitals 
are recorded while the remaining EHR attributes are kept null (following an ideal 
traditional relational model approach). This presence of large number of null values 
leads to wastage of storage space. 

Volatility: Healthcare is an evolving domain. Naturally, many new clinical 
concepts come into existence every year. With this ever-changing environment, 
there is a need for adopting a system that is flexible enough to accommodate the 
changes. 


2 Storing EHRs: Achieving Standardization 


Different healthcare organizations adopt different modes for storing their health 
records. Some organizations store their records on paper, while increasingly many 
are doing so digitally. With the shift to digital medium, EHRs now face the problem 
of standardization. Healthcare is a diverse domain in terms of terminologies and 
ontologies. Standards attempt to define common terminology and ontology for all 
with no ambiguity. Following a standard can provide a common unambiguous 
picture of the data to everyone irrespective of their location and language. Hence, 
EHRs must be standardized, incorporating semantic interoperability. They must be 
designed to capture relevant clinical data using standardized data definitions and 
standardized quality measures. 


2.1 Existing Standards 


Several standards offer guidelines which can be used to build a common format for 
EHR data. Some examples are openEHR Foundation [15]; Consolidated Health 
Informatics Initiative (CHI) [16]; Certification Commission for Healthcare Infor- 
mation Technology (CCHIT) [16]; Healthcare Information and Management Sys- 
tems Society (HIMSS) [16]; International Organization for Standardization (ISO), 
American National Standards Institute (ANSI) [16]; Canada Health Infoway [16]; 
Health Level Seven International (HL7) [17]; ISO 13606 [18, 19]; and European 
Committee for standardization (CEN’s TC 251) [20]. 

Standardized terminologies are required for healthcare applications to ensure 
accurate diagnosis and treatment. For example, SNOMED CT is one of the 
developing standards. 

Recently, Health and Human Services (HSS) of United States favored adoption 
of standards for easy consumer access and information sharing without information 
blocking [21]. 
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2.2 Dual-Model Approach 


Dual-model [22] approach is proposed with the aim of separating information 
domain from knowledge domain. Dual-model approach divides the architecture of 
EHRs into two layers, namely, the reference model (RM) [22] and the archetype 
model (AM) [22]. The presence of the two layers provides flexibility of adding new 
information to the existing EHR system without affecting the existing clinical 
information system. Figure 2 shows the system implementing the dual-model 
approach. This approach segregates the responsibilities of an information tech- 
nology (IT) expert from those of a domain expert. The IT expert addresses various 
data management and interoperability issues in RM. The clinical expert does not 
participate in software development process and provides clinical domain knowl- 
edge which is used by system during runtime for end users (such as, patients, 
doctors, nurses, and researchers). An analogy to help us understand the dual-model 
approach is Lego bricks. Lego bricks are toy plastic bricks (analogy RM) with studs 
which can be connected to other plastic bricks and used to construct various desired 
structures such as, buildings and vehicles (analogy AM). Similarly, in dual-model 
approach, RM defines the fixed part of EHRs system such as, data structures being 
adopted in clinical information system. AM adopts RM for defining the constraints 
related to the underlying medical concepts such as, the name of an attribute, its 
range, its unit, and the data structure that it follows. For example, the clinical 
concept blood pressure (BP) constitutes five attributes (systolic, diastolic, mean 
arterial, pulse pressure, and comments) among which four record various types of 
pressure and one records comments. Various types of pressures recorded follow the 
‘QUANTITY’ data type (for quantifiable or measurable data), while the comment 
attribute follows “TEXT” data type (for textual data). Semantics of ‘QUANTITY’ 
and ‘TEXT are well defined in RM by the IT experts. 
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Fig. 2 System implementing dual-model approach 
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The clinical expert will only adopt the information defined in RM to build the 
medical concept based on their healthcare domain knowledge. Any modification in 
the future can be easily incorporated in AM without modifying RM. For example, a 
new type of pressure of data type ‘QUANTITY’ can be included without modifying 
the existing RM. AM delivers the medical concepts in the form of archetypes which 
are maintained in an archetype repository (as shown in Fig. 2). One of the 
online international standardized repositories is Clinical Knowledge Manager 
(CKM) [23] where archetypes based on the openEHR standard [15] are freely 
available for downloading. An archetype may logically include other archetypes, 
and/or may be a specialization of another archetype. Thus, they are flexible and 
vary in form. In terms of scope, they are general-purpose and reusable [1]. The 
clinical expert does not need to gather technical details about the classes defined in 
RM. User composability is provided by tools such as, archetype editor [24]. This 
enables the user, say, the clinical expert, to build archetype using the drag and drop 
functionality. 

The ultimate goal of the dual-model approach is to bridge the IT expert and the 
healthcare domain specialist communities to foster interdisciplinary partner- 
ship. EHRs system based on the dual-model approach have the capability to 
incorporate new standardized data elements in a timely manner. With the 
dual-model approach, a small schema is defined in RM that can allow any type of 
domain knowledge in AM. As the domain knowledge expands, the archetype can 
accommodate it easily by versioning or creating new archetype using Archetype 
Definition Language (ADL) [25] following the same small schema (defined in RM). 
Once the archetypes are defined, they can be used for adopting the required stan- 
dards. Thereafter, a data model must be selected to handle the issues of sparseness 
and volatility. 


3 Managing EHRs: Dealing with Sparseness 
and Volatility 


Considering the storage of EHRs, the most adopted data model is the traditional 
relational approach. Following the relational model, it requires each and every 
attribute (from different departments including administration, nursing, clinical, 
laboratory, radiology, and pharmacy) of a clinical information system to be stored. 
In real scenario, a patient episode at a medical facility will actually need a very 
small portion of the total number of attributes to be recorded. One reason for this 
may be that the patient may be diagnosed without any laboratory test, and thus, 
there is no data related to lab test would be recorded. When only small portions of 
attributes are recorded, the remaining unrecorded attributes are kept as null. This 
leads to high sparseness in EHR data. A relational model is not capable of dealing 
with sparseness efficiently. 


208 S. Batra and S. Sachdeva 


One more problem with the relational model is volatility, i.e., the capability to 
add new attributes in an existing database schema. Addition of new attributes in a 
relational model is possible but upper limits on the total number of attributes are 
defined by the RDBMS (for example, maximum number of columns in DB2 and 
Oracle is 1012 [26]). 

To deal with sparseness and volatility, data models (EAV, dynamic tables, 
OEAV, and OCOM) other than the relational model need to be adopted as the 
storing standard. 


3.1 Entity Attribute Value Model 


The Entity Attribute Value (EAV) model [27] stores data in the form of triplets 
comprising Entity, Attribute, and Value. Each data entry of relational model is 
stored as one row in EAV with the ‘Entity’ column storing the primary attribute. 
The ‘Attribute’ column stores the attribute name for which the data is being stored, 
and the ‘Value’ column stores the data value. The EAV model stores only non-null 
values of relational model. 

To gain knowledge about the null values, EAV is supported with a metadata 
table (stored separately in the database) which stores information regarding the 
schema of the corresponding relational model. Figure 3a represents a typical rela- 
tional model and Fig. 3b shows the equivalent EAV model. As shown in Fig. 3a, b, 
EAV model creates one row for each column value. Row 1 of the example in 
Fig. 3a is split into three rows in Fig. 3b each containing the value of one non-null 
attribute. 
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Advantages of EAV model: EAV model was proposed as an alternative for 
relational model to provide the following functionalities: 


No Sparsity: EAV model reserves space only for non-null values. When infor- 
mation regarding null values is required, it can be accessed using metadata. 

Volatility: It refers to the flexibility of adding new information to the existing 
knowledge domain. This flexibility is achievable in EAV by simply adding the new 
attributes name in the ‘Attribute’ column of EAV model. 


Disadvantages of EAV model: A rigorous survey by the authors [28] found that 
the EAV model lacks certain aspects of the relational model as described below: 


End user understanding: As depicted in Fig. 3b, gathering information from 
EAV model is complex when compared to a relational model. In relational model 
all information regarding an entity is stored in one row, whereas in EAV model it is 
scattered over multiple rows. 

Query tools’ unavailability: Till date, no query tools are available for managing 
EAV-based data. To compensate, existing RDBMSs are used for building, query- 
ing, and managing EAV-based data. 

Handling multiple data types: The presence of a single ‘Value’ column in EAV 
model restricts the entry to one type of data. To store multiple data types, either 
multiple ‘Value’ columns are required, or multiple EAV tables (each storing a 
different data type) are required. 

Retrieving data: Searching data related to an attribute or to an entity requires an 
exhaustive search of the whole EAV database. Thus, time taken to retrieve infor- 
mation from EAV database is much higher. 

Size expansion with increase in non-null values: If a relational model has k non-null 
attributes for an entity, then the corresponding EAV model will contain k rows (one for 
each attribute) containing three entries each (Entity, attribute, and value). Hence, 
EAV-based storage will consume more space with increase in non-null entries. 


3.2 Dynamic Tables 


Dynamic table is an extension of decomposition model proposed by Copeland and 
Khoshafian [29, 30]. It works by dividing the n-ary relational model (i.e., con- 
taining n attributes) into n tables (as shown in Fig. 3c). Among these n tables, n — 1 
tables are composed of two columns; first for the primary key (“oid”) and second 
for a particular attribute other than the primary key. In addition to these n — | tables, 
there is one more table which contains only one column dedicated to store the 
contents of the primary key as in a relational model. The sole purpose of this 
additional table is to identify the existence of a particular entity in the database. 

Advantages of Dynamic Tables: Dynamic tables overcome the problem of 
sparseness and volatility like the EAV model. In addition, dynamic tables are, in 
fact, superior to EAV because of the following advantages: 
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Retrieving attribute data: Searching information regarding an attribute requires 
access to the corresponding attribute table only. Time complexity of an attribute 
centric query in EAV model is O(1). 

Handling multiple data types: Different tables formed due to partitioning during 
the creation of dynamic tables store different attributes, and are thus, capable of 
handling different data types. 

Size expansion with increase in non-null values: Dynamic tables also expand 
with the expansion of non-null values, but this expansion is less as compared to 
EAV. 

End user understanding: Understanding dynamic tables is easier than EAV 
because they store one attribute in one table with the primary key associated with it. 


Disadvantages of Dynamic Tables: The authors identified some limitations in 
dynamic tables which are described as follows [28]: 


Retrieving a particular entity’s data: It can be easily identified if a particular 
entity data is being stored in dynamic table or not by just searching for its id in oid 
table. However, to search the data corresponding to an existing entity (present in oid 
table) in dynamic tables is more complex since all tables (partitioned for dynamic 
tables) must be searched. 

Memory consumption: Dynamic tables maintain only two entries per attribute 
value (the primary key and the attribute value), whereas EAV maintains three 
entries per attribute value (the primary key, the attribute name, and the attribute 
value). Hence, it may seem that storage requirements for dynamic tables are less 
than that of EAV, but that is not true. This fact was proven experimentally by the 
authors in [28]. More memory consumption in dynamic tables was due to the 
storage overhead of various tables (in dynamic tables) in comparison to just one 
table (in EAV). 


3.3 Optimized Entity Attribute Value Model 


Paul et al. [31] proposed the Optimized Entity Attribute Value (OEAV) model to 
improve search efficiency that was lacking in the EAV model. Constructing a 
database based on OEAV model first requires the data to be in EAV format. Next, 
attribute dictionary is constructed manually which defines numeric coding scheme 
for the attribute names. The combinations of numeric attributes codes and the 
corresponding values are stored as one single value under one column termed as, 
Attribute Value (AV) using the following steps: 


1. Convert numeric attribute code and value to the equivalent binary representation 
comprising 16 bit each. 

2. Concatenate both 16 bit codes to form a 32 bit code. 

3. Now calculate the decimal equivalent of the 32 bit code. 
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For example, we want to calculate the AV value of (1, 11). According to step 1, 
16 bit codes of (1, 11) are calculated, which is equivalent to (0000000000000001, 
0000000000001011). These two 16 bit codes are combined to form a 32 bit code 
which is QOOOOOOOQOQKK0000 1 OODDO00000001011 (step 2) and its decimal equiva- 
lent is 65547 (step 3). The AV value, 65547 is now used as a replacement of (1, 11). 

Finally, the transformed table is sorted based on the AV column and then the 
primary key (oid column in Fig. 3d). An index table is further created to point the 
first record (in the sorted table) of the corresponding attribute (in the index table). 
All the records corresponding to a particular attribute are found in the neighboring 
records; the 16 most significant digits of numeric codes will have the same value for 
same attribute. So, after sorting, all neighboring elements will have the same most 
significant digits until the records for the next attribute are started (having different 
most significant digits). 


Advantages of OEAV model: In addition to the functionalities (of no sparse- 
ness or volatility) offered by EAV model, OEAV model achieves the following: 


Retrieval of attribute data: Search efficiency of OEAV model is due to the 
presence of the index table. The index table points to the starting address of an 
attribute in the sorted table which makes OEAV a search efficient model. 

Size expansion with increase in non-null values: The growth is less in OEAV 
model as compared to EAV model because in OEAV model only two entries 
(entity, AV) are created for each row, whereas in EAV model it is three (entity, 
attribute, and value). 


Disadvantages of OEAV model: In addition to the limitations (of end user 
understandability, query tools unavailability, and handling multiple datatypes) in EAV 
model, the authors identified that OEAV model has the following limitations [28]: 


Calculation overhead: Storage and retrieval of any information in OEAV require 
coding and decoding of AV column, respectively. 

Insertion, modification, and deletion of records: Insertion of a new record is 
complex in the case of OEAV model, since records are finally sorted based on the 
value of the AV column. For inserting a new record, the database should be sorted 
again after the insertion. Doing so requires shifting of all records. 

Retrieving an entity data: To retrieve the information of an entity, an exhaustive 
search of the whole database is required. This process is the same as it is in the EAV 
model. So, there is no significant improvement in searching data related to par- 
ticular entity. 

Handling multiple data types: The need for numeric coding for AV column 
requires numeric data. Hence, OEAV model is capable of storing only numeric data. 

Dependency on machine: As shown in Fig. 3d, OEAV model constructs an 
index table which contains the pointer (address) to the first record of the corre- 
sponding attribute in the sorted table. If the database is reallocated to another 
machine, new address will be allocated to the OEAV database in the new machine. 
Thus, all pointer values in the index table will need to be reassigned to new address 
values, which makes the database machine dependent. 
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3.4 Optimized Column Oriented Model 


Optimized Column Oriented Model (OCOM) [32] combines the features of 
dynamic tables and OEAV model (shown using dotted arrows in Fig. 3e). OCOM 
starts by dividing the relational table into multiple tables (each containing two 
columns, 1.e., position number and attribute value) as in dynamic tables. After the 
division, each divided table is coded (two columns are coded in one column) using 
the coding mechanism of OEAV model. 


Advantages of OCOM model: Apart from no sparseness and volatility, OCOM 
model shows advantages of both OEAV model and dynamic tables which are as 
follows [28]: 


Memory consumption: OCOM model stores coded values of each entry (using 
position number and attribute value) of a attribute in a separate table. Hence, the 
total space consumed by OCOM is minimum among all the other compared models. 

Size expansion with increase in non-null values: OCOM reserves exactly the 
Same number of entries in a coded format as there were non-null entries. Thus, there 
is no growth in size with increase in non-null values rather the expansion is 
equivalent to the growth in non-null entries. 

Dependency on the machine: Absence of index table (as in the OEAV model) 
makes OCOM model a machine-independent data model. 


Disadvantages of OCOM model: OCOM model shows some limitations which 
are as follows: 


Calculation overhead: Calculation overhead of OEAV model is also involved in 
OCOM model. 

Retrieving data: Retrieval of data in OCOM model is quite complex due to the 
presence of coded values which need to be decoded before data retrieval. 

Handling multiple data types: Similar to OEAV, OCOM also stores only 
numeric data as coding mechanism of OEAV model needs to be applied in OCOM 
model. 

End user understanding: User cannot easily understand the coded format of 
OCOM model. 


4 Modeling EHRs: A Big Data Perspective 


The current section considers the big data issues in the modeling of large-scale EHR 
data, especially when it is not stored using relational model approach. EAV, 
dynamic tables, OEAV, and OCOM model are built with a primary aim of handling 
sparseness and volatility. The costs of EAV, dynamic tables, OEAV, and OCOM 
model are different in terms of data volume, velocity, variety, complex querying, 
complex data relationships, and view level support, as shown in Table 1. 
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Table 1 Comparison of data models from big data perspective 


Big data Data models 


requirements FAV Dynamic tables OEAV OCOM 

Volume Handle Handle Handle Handle 
Sparseness Sparseness sparseness sparseness 

Velocity Slow data Slow entity data Complex data Complex data 
retrieval retrieval retrieval retrieval 


Variety Requires Stores multiple Stores only Stores only 
multiple data types of data numeric data numeric data 
type tables 

Complex Requires Requires little Highly complex | Highly 

query modification in modification in complex 
SQL structure SQL structure 

Complex Maintained Maintained Not handled Not handled 

data using EAV/CR implicitly currently currently 

relationship approach 

View level Require Require JOIN Require PIVOT Require JOIN 


support PIVOT operation operation after operation after 
operation decoding decoding 





EAV, dynamic tables, OEAV, and OCOM model are capable of handling 
Sparseness in data. They can make space for storing more data, and thus, handling 
EHRs’ increase in volume. 

One of the most prominent challenges for adopting EAV, dynamic tables, 
OEAV, and OCOM model is the unavailability of query tools. Adopting existing 
RDBMS for these models requires modifications at the logical level, which makes 
data accessing complex. 

Managing the velocity of accessing data becomes complex in case of OEAV and 
OCOM due to the requirement of decoding data, whereas it is simpler in case of 
EAV than OEAV and OCOM, but is still slow. The most efficient model in terms of 
velocity among the discussed models is dynamic tables, which can access attribute 
data very fast, yet is slow for entity-related data. 

Designing of EAV schema supports only one type of data that hinders the 
variety aspect of EHRs. OEAV and OCOM are designed only for numeric data 
which also have same drawback. Only dynamic tables satisfy the variety aspect of 
EHRs. The most important aspects that need to be handled by EAV, dynamic 
tables, OEAV, and OCOM models are described below. 


4.1 Complex Query Requirement 


Query must provide the most efficient means to gain access to a database. It allows 
for coding as needed by a user to gain access to a portion of database in a specified 
condition. Queries in healthcare domain are not only restricted to patient-related 
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information (1.e., clinical queries) but may also extend to those related to population 
(i.e., epidemiological queries). The main reason for using relational model over 
other data models is the absence of query support. Many relational database 
management systems (RDBMSs) available in the market support relational models 
but none exists for EAV, dynamic tables, OEAV, and OCOM. To use a RDBMS 
for querying a database stored as EAV, dynamic tables, OEAV, and OCOM, the 
SQL query structure needs to be modified. For example, to access a particular 
attribute in relational model and the equivalent EAV model, the following queries 
will be used. 


In Relational Model 

select < attribute_name > from < table_name >; 

In EAV Model 

select value from < EAV_table_name > where attribute = “<attribute_name>”’; 


4.2 Complex Data Relationship 


Relationships among data in RDBMS is maintained using referential constraint, 1.e., 
by combination of primary key and foreign key. In case of EAV, the primary key 
concept makes it difficult due to replication of the entity identifier in the ‘Entity’ 
column. To maintain relationships in EAV model, an extension to EAV called 
entity—attribute—value/class relationship (EA V/CR) [33] was proposed. In EAV/CR 
model, relationships among the classes are maintained using either the concept of 
primary and foreign keys or through the use of a special table that is used only for 
maintaining relationships. The main weakness of the EAV/CR model is the 
requirement for high data storage which can lead to problems such as, memory 
shortage and network congestion when applied on worldwide level to achieve 
semantic interoperability. A similar approach can be proposed for OEA V and OCOM. 
Dynamic tables support referential integrity constraint (foreign key) of the relational 
approach. So, there is no need of making any amendments to dynamic tables. 


4.3 View Level Requirements 


Users of databases are comfortable with using relational model where data related 
to a particular entity appears in one row and data related to one attribute in one 
column. With relational model, the user can have direct and simple interpretation of 
data. This is unlike the scenario of a database stored as EAV, dynamic tables, 
OEAV, or OCOM. For instance, to access data related to particular entity or 
attribute, the whole EAV database needs to be accessed. 

To address the issue of complexity at view level of the RDBMS for EAV data, it 
should be converted into the relational model. The conversion of EAV into 
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relational model can be achieved thorough pivoting [34]. However, pivoting 
involves the JOIN operation which has to be performed on different views of EAV 
database. Since join is a costly operation, it also makes pivoting very costly. 

To make dynamic tables more understandable, all divisions (multiple tables) 
need to be joined back. Similar processes of pivoting and joining has to be applied 
to OEAV and OCOM after decoding the values stored therein. 

The process to enable end user understanding could be of concern when the 
whole database needs to be accessed. In realistic scenario, the end user is authorized 
only to view a very small portion of the whole database. Hence, pivoting and 
joining can be applied accordingly, on a very small portion of the database, to 
ensure the positive aspects of the adopted model (EAV, dynamic tables, OEAV, or 
OCOM) while allowing significant end user understanding. 


5 Applying Analytics on EHRs 


The big data model provides a framework for managing data resources, and 
specifies the underlying architecture which can help in extracting knowledge from 
EHRs data at a reasonable computing expense. The architecture presented in Fig. 4 
provides a visual mechanism to manage standardized EHRs for optimizing memory 
use (Volume), faster access speed (Velocity), and storing heterogeneous data 
(Variety). In particular, the management of standardized EHRs from a big data 
perspective is divided into four layers (Fig. 4). 

EHRs constitute wide variety of data including structured data, unstructured 
clinical notes, clinical images, genetic data, epidemiology, and behavioral data 
originated from various departments such as, nursing, imaging lab, pathology labs, 
and the local hospital, as shown in Layer 1 of Fig. 4. EHRs collected from different 
sources are integrated for better understanding of the clinical situation. The 
large-scale EHR data follow different storage formats that can hinder semantic 
interoperability. To achieve semantic interoperability, standardization is essential. 

Layer 2 of the architecture presents the way standards can be adopted for EHRs 
using dual-model approach. Reference model (RM) of the dual-model approach 
specified the stable part of clinical information system designed by IT experts. 
Based on RM, the dynamic part of clinical information system is constructed called 
archetype model (AM). Here, dual-model approach is described using the analogy 
of lego blocks. Lego blocks are produced as basic shape blocks (as in RM) using 
which any complex shape (as in AM) can be build. Any modification to the 
complex shapes built can be achieved with the existing basic blocks. 

EHRs are highly sparse, which drives the choice for the suitable storage model. 
For handling data volume efficiently for highly sparse and volatile EHRs, relational 
model is not well suited. Layer #3 of Fig. 4 presents storage options other than a 
relational model, such as, EAV, dynamic tables, OEAV, and OCOM, which are 
designed with the aim of handling sparseness and adopting any change to the 
existing schema more flexibly. 


216 S. Batra and S. Sachdeva 


= 
Evaluation & 
Presentation 
| } — >| Knowledge 
10 | a 


Patterns 









Layer #3: Modelling Standardized EHRs 













Archetypes 


= 
ety — i 


and Templates 
Clinical User 





Clinical Domain (Dynamic) 


Ewpert 








Reference 
Model (Stable) 
Software 
Expert 
Layer #2: Storing EHRs based on Standards 









Health Inf : 
Location 












Local 
Hospital 


=j <P 
Medical Other data : i 
Imaging Epidemlology E 
data & Behavioural 
Large 
Hospitals 






Layer #1: Integrating Large Scale Heterogeneous EHRs 


Fig. 4 Architecture for standardized EHRs management 


Finally, we must have the required analytics applications that preserve these 
models, so that the models are able to display and store the latest data. 

Data mining (shown in Layer #4) is a very useful process to discover patterns 
that lie hidden otherwise. Many tools such as, WEKA [35] are available for mining. 
Mining data in relational model is easy as the tools available in market are readily 
applicable. Currently, no major tools are available for mining EAV, dynamic tables, 
OEAV, and OCOM modeled data. 

To mine databases using EAV, dynamic tables, OEAV, and OCOM models, the 
data should be converted into the relational format after which standard mining 
tools can be used easily for data analytics. Converting a whole database back into 
relational remains a matter of concern. In a realistic scenario, only a portion of data 
satisfying a particular condition may be required for mining. For example, identi- 
fication of the relationship of high blood pressure patient with diabetes problem 
may require data only corresponding to the high blood pressure patients. So, the 
cost of pivoting may be compensated as only a small portion of data is converted 
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back into relational model for mining. A detailed architecture for preparing stan- 
dardized data for the purpose of mining is presented in [36]. 


6 Summary and Conclusion 


Data is fastly becoming very large for many domains. Healthcare data is big not 
only because of its volume, but also due to its complex, diverse, and temporal 
nature. EHRs as big data require that the 3Vs must be handled carefully along with 
the issues of standardization, high sparsity, and volatility. This chapter explored 
various issues in the EHR domain. In doing so, it was found that the traditional 
relational model is not a suitable option for storing standardized EHRs. 

Data models such as, EAV, dynamic tables, OEAV, and OCOM try to overcome 
the shortcomings of relational model with respect to the EHR requirements. These 
data models provide functionality of no sparseness and volatility for standardized 
EHRs, but they still lack in various aspects, among which the most common is 
entity centric query, query tool unavailability, and end user understandability. To 
improve end user understandability, pivoting and joining mechanisms need to be 
adopted to bring a small and selective data portion (as acquired data by the end 
user) to a better interpreted format (the relational model). Pivoting/joining is also 
required when mining and analytics are to be performed on EAV, dynamic tables, 
OEAV, and OCOM modeled databases. Analytics performed on EHRs can be 
useful for various users (such as, pharmaceutical industry experts, providers, doc- 
tors, insurance experts, researchers, and customers). It may lead to more efficient 
decision support while reducing overall decision-making errors and healthcare cost. 
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Microbiome Data Mining for Microbial 
Interactions and Relationships 


Xingpeng Jiang and Xiaohua Hu 


Abstract The study of how microbial species coexist and interact in a 
host-associated environment or a natural environment is crucial to advance basic 
microbiology science and the understanding of human health and diseases. 
Researchers have started to infer common interspecies interactions and species— 
phenotype relations such as competitive and cooperative interactions leveraging to 
big microbiome data. These endeavors have facilitated the discovery of previously 
unknown principles of microbial world and expedited the understanding of the 
disease mechanism. In this review, we will summarize current computational efforts 
in microbiome data mining for discovering microbial interactions and relationships 
including dimension reduction and data visualization, association analysis, micro- 
bial network reconstruction, as well as dynamic modeling and simulations. 


1 Introduction 


There are two fundamental research questions in microbial ecology [1]. The first 
question is how many kinds of microbes are in a microbial environment? It is 
estimated that there are about 1000 different kinds of bacteria in a human gut; 
however, it is very hard to know what kind of bacteria exist there [2]. The second 
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question is what kind of metabolic functions are the major ones and how microbes 
collaborate to complete a complex biological function? To answer these funda- 
mental questions, we need to study the structure and the function of genes and 
microbes in a microbial community. 

Metagenomic sequencing that investigate the overall DNA content of a micro- 
bial community by high-throughput sequencing technologies without culture any 
individual species has become a powerful tool for microbiome research [3]. It 
allows researchers to characterize the composition and variation of species across 
environmental samples, and to accumulate a huge amount of data which make it 
feasible to infer the complex principle of microbial ecology (Fig. 1). 

Microbiome sequencing can be used to investigate not only the diversity of 
microorganisms but also that of gene functions. For example, functional profiles 
including metabolic profiles [5], protein family profile [6], or taxonomic profiles 
which include operation taxonomic unit table [7], genus, and species profile [8]. 
These profiles indicate the abundance of functional or taxonomic categorizations in 
a microbiome sample which can be represented by a matrix or a tensor. In a profile 
matrix, entries indicate the abundance of a function or taxa (See Fig. 2). Investi- 
gating the profile can help us to answer the two fundamental questions. Interdis- 
ciplinary methods from mathematics, statistics, and pattern recognition could help 
to identify the hidden structure and patterns embedded in a microbial community 
and discover nonlinear relationships between microorganisms and environmental or 
physiological variables (Fig. 2). 

To state the two fundamental questions in microbial community clearly, we also 
have to know how different species coexist and interact in a natural environment. 
Microbiology is experiencing a “holistic thinking” trend that emphasizes the 
interactions of different elements in a microbial community using network analysis 
and modeling tools (at the bottom of Fig. 2). Microbial interaction network (MIN, 
e.g., species—species interaction network) shapes the structure of a microbial 
community and hence forms its ecosystem function and principle such as the 
regulation of microbe-mediated biogeochemical processes. Deciphering inter- 
species interaction is challenging by wet lab due to the difficulty of co-culture 
experiments and the complicate types of species interactions. Researchers have 
started to infer pairwise interspecies interactions such as competitive and cooper- 
ative interactions leveraging to large-scale microbiome data including metagenomes 
[2, 9, 10], microbial genomes [11-13], and literature data [14]. These efforts have 
facilitated the discovery of previously unknown principles of MIN, verified the 
consistency, and resolved the contradiction of the application of macroscopic 
ecological theory in microscopic ecology. 

In this review, we will investigate computational studies in big microbiome data 
mining. We will emphasize computational methods in a unit framework as shown 
in Fig. 2 which are related to dimension reduction and data visualization as well as 
microbial network construction and simulation. 
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Fig. 2 A data mining framework for identifying and analyzing interactions and relationship from 
microbiome data 
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2 Dimension Reduction and Visualization of Microbiome 
Data 


2.1 Dimension Reduction and Data Visualization 
of Microbiome Profiles 


Dimension reduction and data visualization converts the high-dimensional dataset 
X ={Xx1,X2, ...,X,} into two- or three-dimensional data Y= {y1, y2, ...,y,} that 
can be displayed in a scatter plot at a computer screen for exploring data conve- 
niently. The study of microbial ecosystem biology, some classical statistical, and 
pattern recognition methods for data dimension reduction and visualization has 
been applied to analyze microbiomic data [1], such as principal component analysis 
(PCA) [15], multidimensional scaling analysis (MDS) [16], nonnegative matrix 
factorization (NMF) [6, 17], and probabilistic topic model (PTM) [18], (we will 
discuss PTM in detail in the next subsection). Arumugam et al. [19] found that the 
type of microbial compositions in a human gut is not random and it can be classified 
into at least three enterotypes by integrating PCA and clustering analysis. The 
mechanism of enterotypes is still unknown but it may relate the action of human 
immunity systems to bad or good bacteria, or to different methods of excrete wastes 
from the body. Similar to blood type, enterotype is not related to demographic 
factors such as country, age, gender, race, or other body index. Metric-MDS is one 
subclass of MDS, which is also called principal coordinate analysis (PCoA) [19]. 
After combining with phylogenetic distance, PCoA is widely used in the visual- 
ization and interpretation of microbiomic data [20-26]. For instance, the integration 
of PCoA with unweighted UniFrac distance has been used in analyzing the age and 
geographic distributions of 531 human microbiomic [20]. 

NME framework has been used in analyzing metagenomic profiles to gain a 
different and complementary perspective on the relationships among functions, 
environment, and biogeography of global ocean [6, 17]. To investigate if the 
metagenomic dataset is better modeled by manifold rather than Euclidean space, an 
analysis framework based on a nonlinear dimension reduction method—Isometric 
Feature Mapping (Isomap)—has been proposed to explore the nonlinear structure in 
metagenomic profiles [27]. In contrast to PCA and NMF, Isomap identified non- 
linear structure embedded in the data clearly. The results indicate that the nonlinear 
components are significantly connected to environmental factors. 


2.2 Probabilistic Modeling of Microbiome Data 


Probabilistic topic models (PTM) can be used to build a generative model to 
simulate the generation process of a microbial community based on the taxonomic 
or functional categorization from metagenomes. In this model, a microbial com- 
munity can be regarded as a probabilistic mixing of microbial “topics” rather than a 
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linear combination of “components.” These methods provide a generative mecha- 
nism of microbial community which is helpful in understanding the generating of 
complex microbial ecology. 

Based on the functional elements derived from non-redundant CDs catalog in 
human gut microbiome, the configuration of functional groups in metagenome 
samples can be inferred by probabilistic topic modeling [18, 28]. The generative 
model assumes that a microbiome sample could be generated in a style of writing a 
document. Each microbial sample (assuming that relative abundances of functional 
elements are already obtained by a homology-based approach) can be considered as 
a “document,” which is a mixture of functional groups, while each functional group 
(also known as a “latent topic”) is a weighted mixture of functional elements 
(including taxonomic levels, and indicators of gene orthologous groups and KEGG 
pathway mappings). The functional elements bear an analogy with “words”. Esti- 
mating the probabilistic topic model can uncover the configuration of functional 
groups (the latent topic) in each sample. The results are consistent with the recent 
discoveries in fecal microbiota study of inflammatory bowel disease (IBD) patients 
[2], which demonstrate the effectiveness of probabilistic topic model. 

Another method used Dirichlet multinomial mixture distributions to model 
microbial community [29] which provides model-based alternatives for both clus- 
tering and classification of microbial communities. The success of probabilistic 
models is not surprising because of the probability distribution nature of micro- 
biome profiles, 1.e., the relative abundance profile is a composition of probability 
distribution. 


3 Large-Scale Association Analysis and Microbial 
Network Reconstruction 


Network methods have great potential to discover biological characteristics asso- 
ciated with human health, disease, and environment change. Microbiomics data 
usually has high dimensionality characteristics, namely the number of functional or 
taxonomic classifications is far greater than the number of the number of samples 
(machine learning often referred to as the curse of dimensionality). Algorithms 
including correlation analysis, statistical tests, and complex network approach must 
be corrected or improved in order to obtain a reasonable application in microbiome 
data. Furthermore, microbiomics data has its complex data specialty that provides 
difficulties for current methods to identify true relationships. 
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3.1 Relationship Between Microbiomic Data and Metadata 


Linear correlation or regression methods are also employed to investigate the 
relationships among taxa or functions and their relationships to existing environ- 
mental or physiological data (metadata), such as Pearson correlation and Canonical 
correlations analysis (CCA). CCA has been proposed for investigating the linear 
relationships of environmental factors and functional categorizations in global 
ocean [15, 30, 31]. Canonical correlation analysis (CCA) explores linear correlation 
between the data of the principle dimensions by maximizing the microbial and 
environmental variables, in order to find the most relevant environmental variables 
and functional units. 

Generally speaking, in the practical application facing the curse of dimension- 
ality problem, the direct application of CCA is not feasible; we must first apply 
dimensionality reduction and then use the CCA or the regularization of CCA. 
Literature [30] provided a quantitative analysis of the microbial metabolic pathways 
in global marine environment by CCA, and investigated the environmental 
adaptability to find related metabolic markers (called “CCA fingerprint” in the 
paper). Literature [31] investigated metagenomic features using more environment 
variables, climatic factors, and the relationship between primary productions 
through CCA. It was found that temperature, light, and other climatic factors 
determine the molecular composition of marine biological factors. 


3.2 Microbial Interaction Inference 


The interaction among species is very important to an ecological system function, 
for example, the structure of food web plays a pivotal role in the evolution of the 
macro-ecological environment. Metagenomic and 16S sequencing allows inference 
of large-scale interactions between microbial species [32]. In recent years, com- 
putational methods for microbial network inference are proposed based on the 
co-occurrence pattern or correlations of microbial species group. For instance, 
similarity, relevance and mutual information method can be used to predict the 
simple pairwise species interactions [7]; while regression and rule-based data 
mining methods can be used to predict more complex species interactions. 

After the completion of the interaction prediction, network analysis methods will 
help deep understanding of the microbial structure impact on the ecological envi- 
ronment of the visualization of species interaction network, the extraction of 
complex structures, the structural analysis of network attributes (such as distribution 
of the average path length and modularity, etc.) and the exploratory discovery of the 
relationship between the environment and microbes. 
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Reshef et al. [33] proposed a new nonparametric statistical method used to 
extract the complex relationship between large-scale data, which is based on 
“Maximum Information Coefficient” (MIC) without using prior assumption of 
function types, nonparametric optimization can be found through a complex 
association between variables. MIC is based on the idea that if two variables have a 
relationship, then there should exist a scatter plot mesh of the two variables that 
make most of relationships of the data points concentrated in a few cells of the grid 
medium. By searching for this “optimal” grid computing, MIC can discover the 
Statistical association between variables, and the statistical framework based on 
MIC is called MINE (Maximal information-based nonparametric exploration). 
Reshef et al. compared MINE with other methods and found MINE is more suitable 
for large-scale data on the association mining with fast computing speed. MIC has 
attracted widespread interest and has been successfully applied to the 
high-throughput microbiomics data analysis [34-37]. 

Chaffron et al. used Fisher’s exact test with FDR correction of p value to build a 
large-scale network in a scale of microbial community based on co-occurrence 
patterns of microorganism; many microorganisms associated in network are also 
similar evolutionarily [38]. Zupancic and others used Spearman correlation coef- 
ficient for constructing a network of relationships between intestinal bacteria from 
Amish (Amish) people and used regression analysis to investigate the relationships 
between three bacterial networks (corresponding to the three enterotypes) and 
metabolic phenotypes; they found 22 bacterial species and four OTU related to 
metabolic syndrome [39]. 

Faust et al. analyzed the data from the first stage of the Human Microbiome 
Project—16S rRNA sequence data of 239 individuals with 18 body positions (totally 
4302 samples) to build a global interaction network of microorganisms [40]. The 
method for building interaction network in which the introduction of a extended 
generalized linear models (Generalized Boosted Linear Models, GBLM) models the 
prediction of the target position (target site, ts) with species taxa (target taxa, tt) 
abundance X,,,;,; by the source position (source site, ss) with species taxa (source 
taxa, st) abundance Xst, ss: 


X st, ss Xt, t =X st, ss t > p tt, ts, St, ssXst, sse ( 1 ) 
st 


Also due to the high-dimensional nature of the data, the model is prone to 
over-fitting. Hence, a modified linear model (Boosted Linear Models) is proposed, 
based on the linear regression analysis model-LASSO [41]. The final interaction 
network integrates GBLM and other correlation or similarity measure (Pearson 
correlation, Spearman correlation coefficient, Kullback—Leibler divergence, 
Bray—Curtis distance) resulting species interaction networks. The results showed 
that the co-occurrence of microbial interactions often occur in similar parts of the 
body, which was in line with “the different parts of the body are different niche” 
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hypothesis, while interacting with the exclusion often occurs in between evolu- 
tionarily distant species, so tend competition [41]. 


3.3 Network Reconstruction for Compositional Data 


As mentioned earlier, the taxonomic profile has another very significant feature that 
it belongs to a special data types—composition data [42—44]. Due to the small 
number of species in a larger number of taxa exist, a lot of negative correlation are 
generated by bias in the application of Pearson correlation coefficient on conven- 
tional compositional data, and the effect is known as “Composition effect.” To 
overcome this problem, Friedman et al. systematically investigate the species (or 
functional) diversity effect on the compositional data and found that when the data 
has a very low diversity of species, the composition effect can be ignored, and vice 
versa. Therefore, although the same issues in gene expression data by 
high-throughput sequencing, there is not a number of dominant genes, and thereby 
the compositional effect in data of DNA chips, RNA-seg or ChIP analysis-seq is not 
significant even negligible. Friedman proposed a new computational framework 
(SparCC) to overcome the compositional effect [45]. 
SparCC transforms the original matrix OTU relative to abundance ratio x; by 

Xi 


- =logx; — logx; (2) 
Aj 


Jj = log 
to introduce a new variable y;;, New variable y; has several good statistical prop- 
erties than x;. The results found that SparCC can extract relationships from data 
with high accuracy. Experimental results of SparCC on HMP data show that the 
relationships obtained by Pearson correlations have about 3/4 wrong, and it is 
estimated that about 2/3 of the true relations have not been identified by Pearson 
correlation. This result suggests that a deep understanding of the data can inspire a 
new set of data analysis methods for microbiological analysis and there are still a lot 
of space and challenges for dealing with microbiomic data. 


3.4 Inferring Species Interaction from Time Series Data 


Microbial abundance dynamics along the time axis can be used to explore complex 
interactions among microorganisms. It is important to use time series data for 
understanding the structure and function of a microbial community and its dynamic 
characteristics with the perturbations of the external environment and physiology. 
Current studies confined to use time sequence similarity, or clustering time series 
data for discovering dynamic microbial interactions; these methods do not take full 
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advantage of the time sequences. We have proposed an extension of VAR models 
for species interaction inference [46]. 

Again, due to the high-dimensional nature of microbiomics data, the number of 
samples is far greater than the number of microorganisms; direct interaction 
inference by VAR is not feasible. In our previous studies, we have designed a graph 
Laplacian regularization VAR (GVAR) method for analyzing the human micro- 
biome [46, 47]. We validated the strength on several datasets. We found that our 
approach improves the modeling performance significantly. The experimental 
results indicate that GVAR achieves better performance than the sparse VAR model 
based on elastic net regularization [46, 47]. 


3.5 Data Integration for Inferring Species Interactions 


Data integration will take the extracted interactions and relations from heteroge- 
neous datasets and identify the knowledge gaps in the different knowledge sources. 
The input for the data integration module is interactions (that can be extracted from 
any inferring methods or pipeline), and the output is knowledge that has been 
validated and deemed consistent with existing knowledge. Figure 3 shows a pro- 
posed hierarchical structure in a data integration process. Microbial interactions in 
genus or species level will be inferred depending on the availability of data. 

Using data integration, we can also collect other entities information (e.g., 
species—attributes) and relations (e.g., species-environment and species—disease 
relation [48]) because they are important for biological correlation discovery. 
Statistical machine learning approaches for consistency checking of inferred 
interactions and relations from various resources, validation, and qualitative rea- 
soning on integrated knowledge graph will be implemented. The integrated data 
could provide a comprehensive and hierarchical knowledge graph for many kinds 
of relationships in microbial community study. 


È Genus Integration 


© Species Dataset 1 Dataset 2 Dataset 3 
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Fig. 3 The integration of inferred microbial interactions and relations in a hierarchical structure 
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4 Large-Scale Dynamics Simulation for Microbial 
Community 


To better understand the biological phenomena, it is necessary to study the time 
evolution of microbial interaction to get in-depth understanding of a single microbial 
community. Dynamics analysis for the evolution of microbial interactions can reveal 
the mechanisms and rules of microbial ecosystem. Literature [49] analyzed the 
human intestinal antibiotics trends and revealed the effects of antibiotics on intestinal 
microbial community diversity; Literature [34] studied the dynamics in intestinal 
microbial community of a developing infant. However, microbial system is a large 
system which is hard to simulate. For example, it is estimated that in the microbial 
ecosystem of human gut, there are about 1000 kinds of bacteria with a billion of 
bacteria and more than 5 million genes in more than 6000 orthologous gene family. 


4.1 Dynamic Models for Microbial Interaction Network 


For discovery of the dynamics of microbial communities (Dynamic signature), 
Gerber et al. developed a computational framework based on Bayesian methods 
[50], and apply it to a time series gut microbiomic data under antibiotic treat [49]. In 
this method, OTUs are assigned to a few time series prototypes feature (prototype 
signature) by a probability distribution. This identifies which OUTs are related to 
antibiotics reaction. Mounier and co-workers using the generalized gLV (Gener- 
alized Lotka—Volterra) equations to simulate the interaction of microorganisms in 
fermentation yeast of cheese [51]. 

They predicted three negative interactions between species which have been 
experimentally confirmed. gLV equation is a prey—predator model, the so-called 
“destroy winner” model that can be used in this proposal to simulate the micro-scale 
microbial dynamics. Assuming an environment with N microorganisms, the growth 
rate b; of microorganism X;, the interaction strength a; between X; and X;, consider 
the following promotion LV equation 


Xi) =X b+ E aX) (3) 


where X;(t) is the abundance profile of the species i at time t. GLV equations can be 
used to investigate the stability and chaotic behavior of a microbial environment. 
This model can also analyze the robustness of an ecosystem through the investi- 
gation that the strength of an outside interference on a microbial environment in 
order to determine whether the impact of outside interference reversible. 

In microbial environment, predator is usually a virus or winner bacteria in food 
competent. An increase in the abundance of microbial species also increases the 
probability of its prey by the virus (winner bacteria), making the system stable at 
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large scale but with rapid fluctuations at small scale. Hoffmann and collaborators 
used a modified gLV model to simulate the abundance of marine phage [52] and 
found that the phage—-host community dynamics is in transition between long-term 
low abundance and short prosperous. The model predicts that the abundance of 
phage—host prosperous period in its moment of its risk period (coincides to “destroy 
winner’ model), and then the phage abundance decreased rapidly, after that slows 
down at low levels of abundance to survive until the host arrive the next heyday. 


4.2 Dynamic Modeling for Metabolic Network 
at Community Level 


Flux balance analysis (Flux Balance Analysis, FBA) proposed in 1999 is a math- 
ematical model for metabolic network at genome scale [53]. Compared with tra- 
ditional simulation methods, FBA model has satisfied computing efficiency. FBA 
can be calculated to estimate the metabolic flux steady state within a few seconds 
for moderate scale metabolic networks (>2000 responses). 

The basic idea of FBA is as follows: Consider the predicted maximum yield of a 
target substance while the organisms reach a steady state, and the overall metabolic 
flux distribution within the network should be balanced. FBA can effectively use 
the information of stoichiometric matrix and integrate constraint conditions to find 
the optimal solution of a simulation system [53]. According to the flux of each 
reaction and stoichiometric coefficients, all metabolite quality equations constitute 
simple ordinary differential equations. Set the stoichiometry matrix as S, the amount 
of metabolic flow v, and the system reaches steady state when 


S-v =0. (4) 


Stolyar first analyzed the multi-species metabolic model [54]; the model has two 
simple microbes—Desulfovibrio vulgaris and Methanococcus maripaludis. FBA 
analysis on the simple bacterial community accurately predicted the growth of a 
number of co-culture features, including the ratios of the two species. It suggested 
that the metabolic network reconstruction and stoichiometric analysis can not only 
predict the metabolic flow and growth phenotype for single species but also for 
microbial communities. 

117 genome-scale metabolic network models were published by Feb 2013. In 
microbial ecosystem biology, multi-species FBA will also show its potential. 
Constraint-based modeling (CBM) was already used for the inference of three 
potential interactions [13]: negative, where two species compete for shared 
resources; positive, where metabolites produced by one species are consumed by 
another producing a synergistic co-growth benefit; and neutral, where co-growth 
has no net effect. In a two-species system, the CBM solver aims to explore the type 
of interactions by comparing the total biomass production rate (denoted AB) in the 
pairwise system to the sum of corresponding individual rates recorded in their 
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Fig. 4 A constraint-based 


modeling to model pairwise Vpm,aB = MAX )imeta,B} VBM,m 
interaction Subject to: 
SV =0 


Vimin < vi < Vimax 


v; EV 


individual growth (denoted A + B). The CBM model is defined in Fig. 4, where 
Vgm,m 1S the maximal biomass production rate in a system m, corresponding to 
species A and B. When AB « A + B, A and B have a competitive relationship. 


5 Conclusion 


To comprehensively explore microbial ecosystems and make the most potential of 
public microbiomic data, it is necessary to integrate different types of data in 
different scales and develop novel computational methods to overcome these 
assumptions and consider the microbiomic data properties in the analysis proce- 
dure. Data mining for big microbiome data helps us to discover hidden structures 
and patterns in microbiomic data, to understand the structure and function of 
microbial community, and the interaction between microbial community and its 
host environment. The development of these methods needs interdisciplinary efforts 
in computer science, mathematics, and biology by integrating high-throughput 
sequencing data such as metagenome, 16s rRNA sequencing data, metatran- 
scriptome, meta-metabolism, and high-quality metadata, such as environmental, 
chemical, physical, and geographic data and other ecological, genomics data. 
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A Nonlinear Technique for Analysis 
of Big Data in Neuroscience 


Koel Das and Zoran Nenadic 


Abstract Recent technological advances have paved the way for big data analysis 
in the field of neuroscience. Machine learning techniques can be used effectively 
to explore the relationship between large-scale neural and behavorial data. In this 
chapter, we present a computationally efficient nonlinear technique which can be 
used for big data analysis. We demonstrate the efficacy of our method in the context 
of brain computer interface. Our technique is piecewise linear and computationally 
inexpensive and can be used as an analysis tool to explore any generic big data. 


1 Big Data in Neuroscience 


In the last decade, big data has impacted the way research is carried out in areas 
including genomics [37, 45], proteomics [34, 44] and physics [17]. Neuroscience is a 
late entrant in the field of big-data analysis, and a large number of neuroscience stud- 
ies still involves hypothesis-driven science using small-scale data. However, with the 
introduction of big-data projects like the BRAIN initiative [25] in the United States 
or the Human Brain Mapping initiative [36] in Europe, there has been a gradual 
transition toward big data collection, analysis, and interpretation [1]. 

Technological advances have made it possible to simultaneously record big neural 
data and behavioral measurements, while machine learning and data mining algo- 
rithms are being routinely used to explore the relationship between these data. 
Because of their simplicity and ability to be executed in real time, statistical tech- 
niques based on linear, parametric models are often used for this purpose. In the face 
of large data dimension, many of these methods are hindered by the small sample size 
problem and curse of dimensionality. Nonlinear and/or nonparametric techniques are 
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a viable alternative, although their computational cost may be prohibitive for real- 
time implementation. In this chapter, we present a nonlinear technique suitable for 
analysis of big data in neuroscience. The technique retains computational simplicity 
of linear methods, and is therefore amenable to real-time implementations. 

One of the key steps in performing multivariate analysis of big data is dimen- 
sionality reduction. In this chapter, we focus on the key aspect of computational 
challenges and how dimensionality reduction techniques can be potentially used to 
explore, analyze, and interpret results. More specifically, we analyze the challenges 
posed by high-dimensional neural signals and describe a particular feature extrac- 
tion technique which is well suited for multivariate analysis of big data. We conclude 
by demonstrating the efficacy of the method to decode neural signals in a brain- 
computer interface (BCI) setup. 

Brain—computer interface is a system capable of enabling the interaction of a 
user with its environment without generating any motor output [48, 72]. It may be 
envisioned that future BCI systems will gain a widespread adoption in the fields of 
neuroprosthesis and neurorehabilitation. Researchers have proposed designing cor- 
tical prosthetic devices to aid the millions of people suffering from paralysis, due 
to spinal cord injury (SCI), strokes, or neurological conditions such as Lou Gehrig’s 
disease, cerebral palsy, and multiple sclerosis. The ultimate goal of a BCI-controlled 
prosthetic system is to decode in real time information contained in neural sig- 
nals, and to use this information for control of assistive devices, such as arm [21], 
68] or leg [14, 29] prostheses. In addition, BCI systems have been explored as a 
means for delivering neuro-rehabilitative therapy to those affected by chronic stroke 
[8, 15, 38, 49]. 

Typically, the decoding is facilitated by recording neural correlates of the intended 
task (e.g., movement intentions or movement trajectories), and by accumulating suf- 
ficient data over multiple trials into a training database. Future data can then be com- 
pared against the training database, and the intended task can be decoded. Although 
this article focuses on analyzing big neural data specifically in the context of BCIs, 
the method described here can be potentially useful for a wide range of applications 
relying on big data. 


1.1 Dimenstonality Reduction in Neural Signals 


Neural signals are often high dimensional and therefore require special handling 
including dimensionality reduction followed by feature extraction. When approached 
from linear parametric model standpoint, statistical analysis of high dimensional, 
spatio-temporal neural signals is impeded by the small sample size problem and 
curse of dimensionality. 


1. Small Smaple Size Problem. 
This problem arises when data dimension, n, exceeds the number of samples, N. 
Under these conditions, [19], the data space becomes sparse resulting in sample 
statistics being inaccurate [19, pp. 70], [24]. Consequently, the covariance matri- 
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ces become highly singular, and a straightforward implementation of classical 
techniques such as linear discriminant analysis (LDA) [18], is ruled out. 

2. Curse of Dimensionality. 
The storage and manipulation of big datasets poses a severe computational chal- 
lenge. For example, second-order-statistics-based methods give rise to large 
covariance matrices, whose inversion and spectral decomposition may become 
infeasible even with state-of-the-art computer architectures. 


Efficient feature extraction methods are often employed to circumvent the above- 
mentioned hurdles, whereby low-dimensional informative features are extracted 
from the big data and irrelevant data are discarded as noise. The curse of dimension- 
ality and small sample size problem are typically mitigated in the reduced feature 
space. Several feature extraction techniques have been proposed including classical 
discriminant analysis methods [18, 51] and their variants [35, 69, 76] and proba- 
bilistic measure-based techniques [42, 55]. However, most of the proposed schemes 
cannot resolve the small sample size problem and/or the curse of dimensionality. 
Shrinkage-based discriminant feature extraction (DFE) techniques [9, 56] can effec- 
tively solve the small sample size problem by making the covariance matrix regular, 
but they do not address the problem of curse of dimensionality. Additionally, the 
shrinkage parameter needs to be estimated through expensive procedures [22, 63]. 
Subspace-based decomposition techniques [3, 6, 67, 69, 76, 78] provide alternate 
solution to address the problem of high-dimensional data by discarding noninforma- 
tive subspaces. The most widely used subspace-based approach is to reduce the data 
dimension via principal component analysis (PCA). Kirby and Sirovich [30] were 
perhaps the first researchers to use PCA in the context of human face detection. The 
other notable early approaches using PCA-based data reduction are the eigenface 
technique developed by Turk and Pentland [67] and Fisherface method proposed by 
Belhumeur et al. [3]. Many LDA-based approaches [6, 23, 69, 76-78] have been pro- 
posed in the last decade which address the small sample size problem and curse of 
dimensionality and are especially successful in the context of face recognition. Sev- 
eral kernel-based techniques [57, 75] and manifold-based learning techniques [20, 
53] have also been used efficiently in many applications suffering from small sam- 
ple size problem. However, most of these methods were proposed for applications 
like face/object recognition and when applied directly to high-dimensional, spatio- 
temporal neural data, they did not produce satisfactory results [10, 11]. In addition, 
kernel-based methods are typically computationally demanding, and may not con- 
form to real-time processing demands. Hence, the features in neuroscience have been 
traditionally extracted based on heuristic criteria. 

A common heuristic approach is to separately process the data spatially and tem- 
porally. Data can be analyzed spatially [39, 47, 74] by applying the Laplacian filter, 
followed by temporal processing. The inherent assumption in this approach is that 
space and time are separable, which is likely to be violated in practice giving rise to 
suboptimal performance. 

Temporal processing on the other hand involves spectral decomposition of brain 
data and the utilization of power in various frequency bands (e.g., -band or fp- 
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band [47, 74]). In some applications, such approach could be physically intuitive [52], 
but these ad hoc spectral features produce suboptimal classification of brain signals. 
Furthermore, it could give rise to significant individual difference since some users 
are unable to control the u and #-rhythms [33]. Ranking individual features [59, 
60, 66], or even electrodes [52], according to their individual usefulness, and con- 
structing feature vectors by concatenating few of the most dominant individual fea- 
tures is another popular strategy used in conjunction with the above approaches. The 
major drawback of this approach is that the joint statistical properties of features are 
ignored [7, 31, 65], which may produce suboptimal feature vectors. Elaborate algo- 
rithms for the concatenation of features exist [19], but they are combinatorially com- 
plex [7] and have a limited practical applicability. Another approach to dimensional- 
ity reduction is to perform temporal binning of data, 1.e., replace data over a suitably 
chosen bin with their time-averaged value. This technique is often used in conjunc- 
tion with traditional feature extraction approaches [43, 52]. An obvious weakness of 
this technique is that some information is inevitably lost by binning. Furthermore, the 
estimation of the optimal bin size is a challenging problem and renders this method 
unappealing. Recently data-driven adaptive approaches like common spatial pattern 
(CSP) filtering and its variants [2, 5, 26, 40, 50, 70] have been used successfully 
in many BCI applications. The number of channels used is reduced systematically 
in CSP and this kind of approach has been efficiently used in motor imagery-based 
studies. However, spatial and temporal processing remain separated in CSP and joint 
extraction of spatial and temporal features is not possible. 

In the next section,a principal component analysis-based feature extraction/ 
classification scheme is described. The feature extraction method is capable of deal- 
ing with sparse high-dimensional neural data. The method yields a piecewise lin- 
ear feature subspace and is particularly well suited to hard recognition problems 
where classes are highly overlapped, or in cases where a prominent curvature in data 
renders a projection onto a single linear subspace inadequate. Unlike state-of-the- 
art face recognition techniques, the method performs reasonably well on a variety 
of real-world datasets, ranging from face recognition to bioniformatics and brain 
imaging. 


2 Classwise Principal Component Analysis 


We develop a computationally efficient, locally adaptable feature extraction and clas- 
sification technique which can handle big data suffering from the small sample size 
conditions. Our method uses PCA to reduce data dimension globally, while pre- 
serving the class-specific discriminatory information to facilitate subsequent clas- 
sification. The technique is based on a classwise PCA (CPCA) wherein we reject 
noninformative subspace in the high-dimensional data by using PCA on each class. 
The resulting low-dimensional feature space retains class-specific information and 
is used for classification purpose. 
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Fig. 1 2D examples for (a) (b) 
binary classes where PCA CPCA 

and LDA does not work. The PCA 

black straight lines indicate CPCA PCA,LDA 
optimal subspace by our 
CPCA-based method. LDA 
and PCA subspace are 
denoted in gray dashed dot 
lines. a illustrates Gaussian 
classes with overlapped 
means. b shows bimodal 
uniform classes. All the 
classes have equal priors, 
P(@,) = P(@) 


LDA 


Figure | illustrates a couple of two dimensional (2D) binary class examples where 
the classical PCA and LDA fail, however our technique is able to capture the discrim- 
inatory information. Monte Carlo integration validates our finding; the Bayes error 
is about 40 % using PCA and LDA and 24 % using our method in Fig. la, and 50 % 
and 27 % in Fig. 1b, respectively. 


2.1 Algorithm 


In general, for a c-class problem, our technique gives rise to a family of c sub- 


spaces {S,, S2, ..., So}, so that if data is confined to a low-dimensional manifold 
(see Fig. 2), these subspaces will provide for its piecewise linear approximation. Let 
æ; (i= 1, 2, ..., c) be a class random variable and let S, be the its principal com- 


ponent subspace. Then, the ith class data is generally projected best in S,. The same 
subspace should be optimal for classification purpose too, but this is not necessar- 
ily true and further tests are required. In our case, in order to classify data from @,, 
the subspaces S, (j # i) should be explored as well. The classical PCA on the other 
hand approximates data using a single low-dimensional subspace (Fig. 2e) and does 
not use any class-specific information, for example the manifold curvature. LDA, on 
the contrary, uses classwise information, but results in a linear subspace (Fig. 2f). 
Instead of using complex linear techniques, a simple piecewise linear method may 
be used to efficiently reduce data dimension. 

Our method is implemented through two algorithms. Using training data, the 
first algorithm estimates the piecewise linear subspace and comprises the training 
process. The second algorithm chooses an optimal subspace for feature representa- 
tion and successive classification of an unknown (test) data. 
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Fig.2 a Data from 3 classes, labeled by [] (@,), O (œ) and V (œ), is principally confined to a 2D 
manifold. b Projection of data to the 2D principal component subspace of class œ, € class œ, and 
d class 3. e Projection of data to the global principal component subspace. e Features extracted 
using LDA 
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2.1.1 Training Algorithm 


Algorithm 1 (Training) 





1. begin compute 2%, W, (optional) 

2. initialize i — 0 

2. fori i+] 

3. classwise PCA: compute Wr?“ 

4. augment: E; := [WSA | W,| 

5, discriminant: compute TP™® (optional) 
6. compute F; = E; TPF? 

E until 7 = c 

8. return F}, F,..., F, 

9. end 


2.1.2 Training Algorithm 


Definition of Variables: Let the input be ¥;, a collection of data x belonging to 

class w;. The class mean, u; € R”, and covariance, X, € R, are defined as: u, = 

1/|X;| Dex, x and E, = 1/(|X| — 1) È ex œ- uE- u;)', respectively, where | . | 

denotes the cardinality of a set and n is the dimension of the data vector. The prior 

state probabilities are estimated empirically from the data, i.e., P(@;) = |&,|/ ye Al: 
Firstly, the between-class scatter matrix is computed as 


2, = DIP) (u; — a); - u)" E R” (1) 
i=1 


where P(œ;) is the class probability, „u; is the sample mean of class @,, u is the (over- 
all) sample mean, defined by 


u= > Pœ); E R”. (2) 


c 
i=1 


The matrix W, € R”*f is computed such that its columns form an orthonormal basis 
for the range of 2, defined as R(%,) = {é E R” : é=} 7, WV E€ R”). It follows 
from (1) and (2) that %, has maximum c — | linearly independent columns, therefore 
d < c — 1. The computation W, is optional, and is explained later. 

The algorithm then iterates over classes, and each time the principal compo- 
nent subspace S, is estimated. The m' principal eigenvectors of the class covariance 


x, € R”” forms the columns of the matrix WPCA = RX, which represents a basis 
for S;. The choice of m; is left to the user and several options for retaining number of 
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Fig. 3 Two classes marked 
by O (@,) and O (œ), 
whose principal directions 
are nearly parallel 





eigenvectors/eigenvalues are provided. This includes selecting number of eigenvec- 
tors based on the shape of eigenvalue spectrum, retaining eigenvalues higher than 
the median/mean of nonzero eigenvalues or keeping eigenvalues that capture certain 
percentage of the cumulative energy. In the next step, the matrix E; € RX", +4) is 
formed by concatenating vo and W,. The columns of E; are orthonormalized 
through the Gram-Schmidt procedure resulting in orthogonal projections. Omission 
of step 2.1.1 renders the definition of £; trivial and reduces to £; = W®PSA (formally 
d = 0). W, is included in the algorithm to take into account the cases where discrim- 
inatory information is present in the class means and not in the class covariances. 
One such example is Fig. 3, where the projections of data in the PCA subspace is 
highly overlapped, and the addition of W, results in discriminatory information. 
Typically, m; is so chosen that m! + d <n, and small sample size problem and 
curse of dimensionality are handled by projecting data to a basis defined by E,. 
Following the dimensionality reduction, classical discriminant feature extraction 
(DFE) techniques, such as LDA, can now be applied at this stage (step 2.1.1). The 
use of a DFE technique, although optional, may further boost classification perfor- 
mance. When a linear DFE technique is chosen, the overall feature extraction matrix 
F; € R™ is defined as F; = E; TP™E where TPF e BR": +*"" is a feature extraction 
matrix of the linear DFE method used, and m, is the final dimension of the feature 
space which can be specified by the user. If step 2.1.1 is not used, the feature extrac- 
tion matrix becomes F, = E; (formally ee = I, where J is the identity matrix). Our 
algorithm always yields c feature extraction matrices, whose columns provide bases 
for c subspaces. Examples shown in Fig. 2b—d are obtained by applying Algorithm 1 


without the optional step 2.1.1 and with m! = m, = m, = 2. 
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2.1.3 Optimal Subspace Algorithm 


Let the high dimensional test data be x € R”. Ultimately, x should be classified in one 
out of c subspaces estimated using Algorithm 1. The following algorithm describes 
the procedure of selecting the optimal feature space. 


Algorithm 2 (Optimal Subspace) 





1. begin initialize x —— test vector, F4, F3, ..., /.<— feature extraction matrices, 
i — 0 
2 for i — i+ 1 
3 project ¥ = F] x 
4. initialize j —— 0 
5 for j — j+ 1 
fo) P@,) 
6 Bayes rule: compute P(œ | x’) = ———————— 
: fœ) 
7. until j = c 
8 maximize over classes J(i) = arg max P(œ | x') 
l<j<c J 
9 until į = c 
10. maximize over subspaces I = arg max PO; | x! ) 
1<i<c i) 
11. return I, JD 
12. end 


Algorithm 2 estimates the optimal subspace. The outer loop goes over all the 
c subspaces and in each such subspace, spanned by the columns of F; € R’”, a 
projection, x’ € R”, of x is found. The algorithm then loops over classes, and the 
posterior probability of class œ; given the projection of x is found via the Bayes 
rule for each subspace. This step results in projecting (training) data from each class 
onto the subspace at hand, and estimating the class-conditional probability density 
function (PDF), denoted by f(. | æ). We approximate these PDFs with Gaussian 
densities, i.e., f(. | o) ~ N (F ; Hj, F TZ, F ;). This approach is consistent with Algo- 
rithm 1, which also used second-order sample statistics, namely „u; and 2. Hence, 
this approach is best suited for Gaussian classes, although it should also give good 
results in cases when the discriminatory information between classes lie in the first 
two statistical moments. When the data violates the Gaussian assumption, different 
approaches may be preferable, such as using the mixture density approach [16], or 
the kernel density approach [46]. However, these methods result in a higher com- 
putational cost compared to the Gaussian assumption approach. Irrespective of the 
PDF estimation technique, the mathematical formalism of step 2.1.3 and subsequent 
steps, remains the same. 

The algorithm then computes the maximum a posteriori (MAP) in each subspace. 
In other words, the most likely class is found given the projection of x, and the class 
labels of these locally optimal classes are denoted by J(i). Each subspace may end 
up with a different class having the maximum posterior probability. The final class 
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membership is then found by using the MAP rule to the local winners (step 2.1.3), 
and the winning subspace is marked by I. The MAP rule (step 2.1.3) holds for differ- 
ent dimensional subspaces too since the posterior probabilities are normalized. The 
feature extraction matrix for x is given by the basis F; of the winning subspace and 
test feature, Fr x, to be used for final classification is the projection of the test data x 
onto the chosen subspace. After the feature extraction process is completed, the test 
features can be classified by applying any chosen pattern classifiers. 


2.1.4 Algorithm Summary 


The proposed method is possibly an improvement over single-subspace techniques, 
especially where high-dimensional data lies in a low-dimensional manifold which 
cannot be approximated by a linear subspace. CPCA-based method is expected to 
perform better compared to other techniques on such hard classification problems. 
Moreover, LDA-based methods suffer from the feature subspace dimension con- 
straint, m; <= c — 1, whereas CPCA-based methods are more flexible in the choice 
of m,, thereby giving a better performance. Comparisons of CPCA and other com- 
peting techniques using different datasets are given in [11]. 

While the number of principal components retained in each class may differ over 
classes, the final feature dimension, m,, is kept constant for the sake of simplicity and 
comparison with other methods. The choice of m; can also differ for each class under 
the CPCA framework without changing the mathematical formalism. Using different 
m,, however, might lead to a combinatorial problem and is hence not pursued. 

There are a couple of drawbacks of the proposed method. The time complexity of 
CPCA is O(c) where c is the number of possible subspaces, and hence the method 
is not computationally efficient for large number of classes. However, within a sin- 
gle subspace, the feature extraction scheme is linear and reduces to a simple eigen 
decomposition, which can be implemented efficiently using matrix manipulations. 
The CPU times for different datasets using the proposed method are reported in [11] 
and demonstrate its computational efficiency. While the technique scales unfavor- 
ably with number of classes, c, its performance is comparable to the other methods. 
One other weakness of the proposed method is that by keeping the m, eigenvec- 
tors of the class covariance, ÈX., only the regularity of the class covariance, %,, is 
guaranteed. However, there is no assurance that the other class covariance matri- 
ces, È., (j # i), will remain nonsingular in the subspace S;. A potential solution is to 
shrink S$; (by reducing m) until all the class covariances become regular. Clearly, 1f 
S; C N(%)G # i), where N(È;) is the null space of },, no choice of m, would render 
2, regular in S;, and the method may fail. However, based on the analyzed neural 
datasets and datasets in [11], this scenario seems highly unlikely. Interested readers 
can refer to [11] for detailed analysis of the proposed method. 
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3 Experiments with Big Datasets in Brain—-Computer 
Interface 


The previous section discuss different feature extraction and classification techniques 
that can be used on large-scale data for classification purpose. Now, we shift our 
focus to the application of our methods in the context of BCI. 


3.1 BCI System 


Successful BCI experiments have been reported using both invasive [21, 33, 41, 58, 
62, 71] and noninvasive brain recording technologies [4, 28, 47, 48, 72, 74]. These 
results hold promise for people with severe motor disabilities, such as those suffering 
from amyotropic lateral sclerosis (ALS), brainstem stroke, or other neuromuscular 
diseases. While most of these patients have little, or no muscular control, typically 
their cognitive functions remain unaffected by the paralysis. The goal of BCI is to 
convert these thoughts into actions. While promising experimental results have been 
obtained [4, 21, 33, 41, 48, 58, 62, 71, 73, 74], a fully operational practical BCI 
remains a challenging task. 

In this section we describe a standard memory-reach experimental setup and 
demonstrate the efficacy of CPCA in decoding neural signals. 

Given a set of brain data, the simplest fully operational BCI system must be able 
to answer with certainty at least two questions: 


1. Is the subject’s intention to move or not? 
2. If the movement is intended, should the subject move to the left or right? 


For simplicity, we will refer to these two questions as the when and where questions, 
respectively. Consequently, the BCI system can be modeled as a finite state machine 
(FSM) with Idle, i.e., “do-not-move” state as well as Move state (see Fig. 4). Tran- 
sitions between the states are initiated with each newly acquired block of brain data 








Fig. 4 FSM diagram of a minimal practical BCI system. Allowed state transitions are denoted by 
arrows 
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and are executed in real time. For example, based on the most recent data analysis, 
the system in the /dle state can retain the present state (self-pointing arrow) or make a 
transition to the Move state. It should be noted here that in case of paralyzed patients, 
the Move state should be replaced with Intent to Move state. Also note that the Move 
state consists of two mutually exclusive substates: Left and Right. Clearly, the Left 
and Right substates can be replaced with any other dichotomy, such as “Up/Down,” 
“Yes/No,” “True/False,” and similar. Finally, we note that a BCI system can have 
more than two states (see [61] for example) and/or substates, although the presence 
of additional states will necessarily increase the number of parameters and may result 
in higher error rates. 

Here, we apply our proposed technique to answer the where questions under a 
standard memory-reach experimental setup. In the following sections, we will dis- 
cuss the experiment and results using our method to decode the direction 1.e., left or 
right. Our techniques can serve as a general analytical tool for any large-scale data 
like images or videos. We have also applied our methods for various other applica- 
tions including face recognition and digit recognition, cancer detection, and many 
others (see [11] for details). 


3.2 Direction Estimation 


Decoding directional activities using neural signals has been one of the main focus 
areas of BCI research [10, 41, 47, 52, 54, 61]. BCIs equipped with efficient direc- 
tional tuning capabilities can assist paralyzed patients in performing simple tasks 
using output devices like robotic arm, thus restoring some of the impaired motor 
functions. Furthermore, techniques used for successful estimation of direction can 
serve as analytical tools to understand and analyze spatial processing in the brain 
and provide useful insights related to brain research. In many BCI approaches [12, 
41, 52, 61], subjects are instructed to carry out planned-reach movements while 
their neural activities are recorded. Decoding the direction of such planned-reach 
movements essentially answers the “where” question (Sect. 3.1). Here we use ECoG 
recordings of humans, using standard memory-reach setup, to decode the two pos- 
sible directions (left or right) of target appearing on screen using our proposed 
approaches. In the following sections, we describe the experimental setup for direc- 
tion estimation, followed by results using our methods. Finally we analyze the results 
obtained and discuss the key observations with concluding remarks. 


3.2.1 Experimental Setup 


The performance of our method was tested on a set of ECoG data of a severely 
epileptic patient, adopted from Rizzuto et al. [52], and an EEG dataset of normal 
subjects recorded in our lab. The ECoG signals were recorded from the human brain 
during a standard memory-reach task, consisting of 4 periods: fixation, target, delay, 
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Fig.5 Left Sagital MRI image showing electrode placement in subject. Electrodes were implanted 
orthogonally in a lateral to medial trajectory perpendicular to the intersection of the thin red lines. 
Right Coronal view of depth electrode placement in the same subject. The electrode trajectory is 
shown in red 


and reach. The experiment consisted of the subject making directed arm movements 
to different locations (left or right) on a touchscreen interface (see Figs.5 and 6). 
In the beginning of each trial, a fixation stimulus is presented in the center of the 
touchscreen and the participant initiates the trial by placing his right hand on the 
stimulus. After a short fixation period, a target is flashed on the screen, followed by 
a memory period. The fixation is extinguished after the memory period, which acts 
as a cue for the participants to reach to the memorized location indicated previously 
by the target. The fixation, target and memory periods are varied uniformly between 
1 and 1.3 s. The subject had several electrodes implanted into each of the follow- 
ing target brain areas: orbital frontal (OF) cortex, amygdala (A), hippocampus (H), 
anterior cingulate (AC) cortex, supplementary motor (SM) cortex, and parietal (P) 
cortex. The total number of electrodes implanted in both hemispheres was 91 and 
162 trials were recorded for each period. The signals were amplified, sampled at 200 
Hz, and band-pass filtered. The goal of our study is to predict the label of the trial 
(left vs. right) based on 1 s of data during the four periods. Note that data is a vector 
in 18200-dimensional space (n = 91 x 200). 

For the EEG experiments, a similar setup adapted from [52] was used, with two 
periods: fixation and target (see Fig.7). The EEG signals were acquired using an 
EEG cap (Electro-Cap International, Eaton, OH) with 6 electrodes placed in occipital 
and parietal regions following the international 10—20 system, and the signals were 
amplified, band-pass filtered, and sampled at 200 Hz (Biopac Systems, Goleta, CA). 
The number of trials (left + right) was N = 140 per session, and there were 3 such 
sessions. Our goal is to predict the label of the trial (left vs. right) based on 1s of 
data. 
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3.2.2 Results 


We are decoding the location of target (left/right) from the neural data and the out- 
put of our analyses could potentially be used to decode a person’s motor intentions 
in real time. The ECoG and EEG data were classified using CPCA. The data was 
also decoded using DLDA [76] for comparison purpose. Quadratic classifier was 
used for all decoding results. Principal components with eigenvalues exceeding the 
mean eigenvalue of the covariance matrix for each class were retained in CPCA. 
Leave-one-out cross validation was used for reporting the performance. Using val- 
idatory data to find the final reduced dimension results in some degree of overfit- 
ting. Thus the reported classification rates are not generalizable. The performance 
reported are feasible and suffice for comparison purpose. Leave-one-out CV pro- 
duces nearly unbiased error estimates [32] as compared to k-fold CV and random 
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Table 1 The performances (%) of CPCA and DLDA for ECoG data, during target, delay and reach 
period. (Top) Unconstrained, (bottom) constrained data 





Table 2 The performances (%) of CPCA, DLDA and threshold-based DLDA for EEG data, during 
target period 


Wl) Wwl) NN] |e 


split methods [16]. The performance using leave-one-out CV can be reproduced and 
multiple runs is not required for computing good error estimates. Hence leave-one- 
out CV is best suited for comparing various feature extraction methods. 

The estimated classification rates for the ECoG and EEG datasets are reported in 
Tables 1 and 2, respectively. Additional results with more subjects is illustrated in 
[12, 13]. We make the following observations based on these results. Our CPCA- 
based classification method performs well and produces the best classification in all 
cases. CPCA method sometimes produces an improvement of around 50% when 
compared with DLDA-based method, using both unconstrained and constrained 
data. Typically the first 100-150 ms of the target period can be attributed to visual 
processing delay [64] and can be discarded. Similarly based on brain areas of interest, 
subsets of electrodes can be chosen. 

LDA-based methods perform poorly giving almost chance performance, espe- 
cially for EEG data, whereas our method results in better classification. Only 1-D 
features were used for CPCA to make it comparable to DLDA since LDA-based 
methods can have at most (c — 1)-D subspaces (c = 2 here). CPCA using larger fea- 
ture dimension(e.g., m = 2,3) produced marginally better results. LDA-based tech- 
niques fare poorly on noisy neural data and gives better results when the data is 
comparatively less noisy as in face recognition, object recognition tasks. 
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3.2.3 Discussion 


All analysis reported in the Tables 1 and 2 process the data without assuming space- 
time separability and analyzing the filters produced by these methods, we are able 
to comment on the important brain areas and time instants, which carries discrimi- 
natory information and helps in identifying the two target directions, left, right. The 
spatio-temporal filter has the same dimension as the original data having rows equal 
to the number of electrode channels and columns denoting time. We analyze the 
filters produced by CPCA and discuss some key aspects of these filters. 

Figure 8 shows the the spatio-temporal filters, obtained using CPCA, correspond- 
ing to 1 s of ECoG data during the target period. Note here that we use a piecewise 
linear method and unlike DLDA which generates a single subspace, our technique 
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Fig. 8 Spatio-temporal filters for ECoG data for target period using CPCA. Top figure illustrates 
Subspace 1 and bottom figure shows subspace 2 
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here generates 2 subspaces (one for each class) which acts as a nonlinear manifold 
approximating the high dimensional brain data in the original space. 

The prefix L or R denotes the left and the right hemisphere. Our method produces 
a spatio-temporal filter and there is no need to separately process space and time. 
Most informative brain area and time epoch for decoding the location of the target 
are given by large positive (dark red) and large negative (dark blue) coefficients. Most 
of the information is present ~150—200 ms poststimulus (onset of the target is at 0), 
which is consistent with the visual information processing delay [64]. 

From the distribution of important coefficients in Fig. 8, discriminatory informa- 
tion appears to be localized in time and space, while the noise is distributed, as 
shown by large areas having near-zero coefficients (light blue, green and yellow). In 
Fig. 8, some spatio-temporal areas within the stimulus onset appear 100 ms, which 
is in contradiction to the latency of visual information processing (~150 ms [64]). 
Discriminatory information is not expected so early on in the trial, and these coeffi- 
cients mostly represent noise artifacts, such as biological noise, recording hardware 
noise or ambient noise. 

From Fig. 8, it follows that the Parietal and Hippocampal regions play in impor- 
tant role in distinguishing the two target locations, left and right. The findings are 
consistent with the cortical studies as Hippocampus and Parietal regions are known 
to be associated with the spatial location [27]. Orbitofrontal region (OF), associ- 
ated with decision making, and Supplementary Motor Area (SMA), associated with 
planning movement, also shows up as important regions. Since the target state is fol- 
lowed by the delay state, where one would expect OF and SMA to exhibit prominent 
contribution, the finding is not surprising. 

It can be concluded based on the results that our proposed method efficiently 
decodes large-scale neural data, suffering from small sample size problem. We have 
successfully applied our technique to decode direction of planned-reach movements 
in human using ECoG and EEG recordings. Space-time separability assumption is 
not required in our method and it can be used as an analysis tool to explore the any 
generic big data. 

With the advent of large-scale recording technologies and collaborative data col- 
lection efforts in neuroscience, efficient dimensionality reduction techniques will be 
crucial to the big data analysis. Only time will tell the effect big data has on our under- 
standing of neural processes and the insights it provide. We hope that we can solve 
the challenges related to big data analysis and it provides us with thought-provoking 
results in the coming days. 
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Big Data and Cancer Research 


Binay Panda 


1 Introduction 


The advent of high-throughput technology has revolutionized biological sciences in 
the last two decades enabling experiments on the whole genome scale. Data from 
such large-scale experiments are interpreted at system’s level to understand the 
interplay among genome, transcriptome, epigenome, proteome, metabolome, and 
regulome. This has enhanced our ability to study disease systems, and the interplay 
between molecular data with clinical and epidemiological data, with habits, diet, 
and environment. A disproportionate amount of data has been generated in the last 
5 years on disease genomes, especially using tumor tissues from different subsites, 
using high-throughput sequencing (HTS) instruments. Before elaborating the use of 
HTS technology in generating cancer-related data, it is important to describe briefly 
the history of DNA sequencing and the revolution of second and third generation of 
DNA sequencers that resulted in much of today’s data deluge. 


2 Sequencing Revolution 


The history of DNA sequencing goes back to the late 1970s when Maxam and 
Gilbert [1] and Sanger, Nicklen and Coulson [2] independently showed that a 
stretch of DNA can be sequenced either by using chemical modification method or 
by chain termination method using di-deoxy nucleotides, respectively. Maxam and 
Gilbert’s method of DNA sequencing did not gain popularity due to the usage of 
toxic chemicals and the di-deoxy chain termination method proposed by Professor 
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Fred Sanger became the de facto standard and method of choice for researchers 
working in the field of DNA sequencing. Many of the present-day high-throughput 
next-generation sequencing methods (described later) use the same principle of 
sequencing-by-synthesis originally proposed by Sanger. The pace, ease, and 
automation of the process have since grown further with the advent of PCR and 
other incremental, yet significant, discoveries including introduction of error-free, 
high fidelity enzymes, use of modified nucleotides, and better optical detection 
devices. It is essentially the same technology, first proposed and used by Fred 
Sanger [2], with modifications that led to the completion of the first draft of the 
Human Genome Project [3, 4] that ushered in a new era of DNA sequencing. 
The idea behind some of the first generation high-throughput sequencing 
(HTS) assays was to take a known chemistry (predominantly the Sanger’s 
sequencing-by-synthesis chemistry) and parallelize the assay to read hundreds of 
millions of growing chains of DNA rather than tens or hundreds as done with 
capillary Sanger sequencing. The processes for HTS comprise mainly of four dis- 
tinct steps, template preparation, sequencing, image capture, and data analysis 
(Fig. 1). Different HTS platforms use different template preparation methods, 
chemistry to sequence DNA, and imaging technology that result in differences in 
throughput, accuracy, and running costs among platforms. As most imaging 
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Fig. 1 Steps involved in HTS assays involving cancer patient samples and variant discovery, 
validation and interpretation 
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systems are not designed to detect single fluorescent events, clonal amplification of 
templates prior to imaging is incorporated as a part of template preparation before 
optical reading of the signal. In some cases, as in the case of single molecule 
sequencing, templates are not amplified but read directly to give base-level infor- 
mation. Some platforms are better suited than others for certain types of biological 
applications [5]. For discovery of actionable variants in tumors, accuracy is more 
important over all other parameters. Therefore, some HTS platforms are better 
suited to study tumor genomes over others. However, as the cost per base goes 
down, accuracy is increasingly achieved by higher coverage, thereby compensating 
errors with higher number of overlapping reads. The first publications on the human 
genome resequencing using the HTS system appeared in 2008 using pyrose- 
quencing [6] and sequencing-by-synthesis using reversible terminator chemistry 
[7]. Since that time, the field that has gained the most amount of information using 
HTS platforms is cancer science. The discovery of novel DNA sequence variants in 
multiple cancer types using HTS platforms along with the advances in analytical 
methods has enabled us with the tools that have the potential to change the way 
cancer is currently diagnosed, treated, and managed. 


3 Primary Data Generation in Cancer Studies 


Various steps involved in a typical high-throughput experiment involving cancer 
tissue are depicted in Fig. 1. Briefly, when the patient is admitted in the hospital, 
clinical, epidemiological and information on habits and previous diagnosis, and 
treatment (if any) is recorded. Any study involving human subjects must be 
preapproved by an institutional review/ethics board with informed consent from all 
participants. Following this, analytes, full history of patients, including information 
on habits, and previous diagnosis/treatment (if any) are collected. Then the patients 
undergo treatment (surgery/chemoradiation) and the tumor tissue is collected and 
stored properly till further use. Once the tumor/adjacent normal/blood is collected, 
nucleic acids are isolated, checked for quality, and used in library/target preparation 
for HTS or microarray experiments. Once the raw data is collected, the data is 
analyzed by computational and statistical means before being integrated with 
clinical and epidemiological features to come up with a set of biomarkers, which is 
then validated in a larger cohort of patients. 


4 High-Throughput Data 


HTS platforms generate terabytes of data per instrument per run per week. For 
example, the Illumina HiSeq 4000 can generate nearly 3 terabytes of data per run in 
7 days (or >400 Gb of data per day). This pose challenges for data storage, anal- 
ysis, sharing, interpreting, and archiving. 
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Although there are many different HTS instruments in the market, the bulk of the 
cancer data so far have been generated using the Illumina’ s sequencing-by-synthesis 
chemistry. Therefore, a detailed description is provided on the data size, types, and 
complexity involved in cancer data generated by the Illumina instruments. Below is 
a description of different data types, usually produced during the course of a cancer 
high-throughput discovery study. 

Despite the fact that the process of high-throughput data generation using Illu- 
mina sequencing instruments has become streamlined, never-the-less, there are 
inherent limitations on the quality of data generated. Some of the limitations are high 
degree of errors in sequencing reads (making some clinical test providers sequence 
up to 1000x coverage or more per nucleotide to attain the requisite accuracy), 
shorter sequencing reads (HiSeq series of instruments do not produce data with 
longer than 150 nt read length), the assay not interrogating the low-complexity 
regions of the genome, and higher per sample cost (to gain the requisite accuracy, 
one needs to spend thousands of dollars per sample even for a small gene panel test). 
Details on different data types generated by Illumina HiSeq instrument, their 
approximate sizes and file type descriptions are provided in Table 1. 


5 Primary Data Analysis 


The cancer data analysis schema is represented in Fig. 2. First, the raw image data 
from the sequencing instruments are converted into fastq format, which is con- 
sidered as the primary data files for all subsequent analysis. Before analyzing the 
data, the quality of the fastq files is checked by using tools like FastQC (http:// 
www.bioinformatics.babraham.ac.uk/projects/fastqc/), or with in-house scripts to 
reduce sequencing quality-related bias in subsequent analysis steps. Next, 
sequencing reads are aligned against a reference sequence. Broadly, the alignment 
tools fall into two major classes, depending on which of the indexing algorithm it 
uses: (hash table-based or Burrows Wheeler transformation (BWT)-based). Some of 
the commonly used alignment tools that use hash table-based approach are Bfast 
[8], Ssaha [9], Smalt [10], Stampy [11] and Novoalign [12] and the ones that are 
BWT-based are Bowtie [13], Bowtie2 [14], and BWA [15]. BWA is the most 
widely used aligner by the research community. Lately, many of the alignment 
programs are made parallel to gain speed [16—19]. Most aligners report the results 
of the alignment in the form of Sequence Alignment/Map (SAM, and its binary 
form the BAM) format [20] that stores different flags associated with each read 
aligned. Before processing the aligned files for calling single (SNVs)—and/or multi 
(indels)—nucleotide variants, copy number variants (CNVs), and other structural 
variants (SVs), a few filtering, and quality checks on the SAM/BAM files are 
performed. These include removal of duplicates and reads mapped at multiple 
locations in the genome, realigning reads with known Indels, and recalibrating base 
quality scores with respect to the known SNVs. Once the SAM/BAM files are 
checked for quality, the files are used for variant calling. Although there are 
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Table 1 Different types of cancer data generated from a typical Illumina sequencing instrument 
and their descriptions. 


Data types Description 
Raw data e Multiple instruments, from multiple vendors and produced by multiple 
scientists 


e Produced as image data and never leaves the primary instrument that drives 
the sequencing instruments 

e Image data (in multi-terabytes) are automatically discarded after the initial QC 
e Image files are converted to base call files (BCL) after the initial QC and 
transferred to a server for further analyses 


Primary e Usable data that is derived from raw data and serves as the first-level primary 
data data for all subsequent analyses 
e Usually in the form of fastq files 
e Usually kept for 3—5 years depending on the project duration and complexity 
before being archived 
e A single fastq file size can vary anywhere between 5—15 Gb (for a 50 MB 
human exome at 50—100X) to 200-350 Gb (for a 30-50X whole human 


genome) 
Secondary e The fastq files are aligned to a reference genome and kept as aligned file 
data (SAM/BAM format) 


e A single SAM/BAM file can vary in size (2—5 GB in size for a single 50 MB 
human exome at 50—100X). 


Tertiary e Files produced after the variant calls are made (in most cases VCF files) 
data e Usually vary in size (40-250 Mb for a 50 MB human exome data with 
50-100X coverage) 
e Kept for selected project-specific files for up to a year and then archived 


Final results e Amalgamated with other data types from multiple instruments 
e Used in publication 
e Between 10Mb - Gb in size depending on the data type for each sample 
e Kept for a very long time 


multiple tools for calling variants, the widely used popular one is the genome 
analysis toolkit (GATK) [21, 22] developed at the Broad Institute, USA. GATK 
implements variant quality score recalibration and posterior probability calculations 
to minimize the false positive rates in the pool of variants called [22]. Variants are 
stored in a file format called variant call format (VCF), which is used by various 
secondary and tertiary analysis tools. Another commonly used file format for cancer 
data analysis is called mutation annotation format (MAP), initially made to analyze 
data coming out from the cancer genome atlas (TCGA) consortium. The MAF 
format lists all the mutations and stores much more information about the variants 
and alignment than the VCF files. 
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Fig. 2 Cancer data analysis schema 


6 Secondary Data Analysis 


Before secondary analysis, usually the PASS variants produced by GATK (standard 
call confidence >= 50) within specific genomic bait (used in exome or gene panels) 
are filtered and taken for further use. Tumor-specific variants are detected by fil- 
tering out the variants found in its corresponding/paired normal sample. During this 
process of calling tumor-specific variants, sequencing reads representing a partic- 
ular variant in a tumor sample that have no corresponding reads in matched normal 
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in the same location are ignored (using the callable filter of the variant caller). Then 
common SNPs (found in the normal population) are filtered out using a list of 
variants found in the databases like dbSNP and 1000 genome project. Therefore, 
only variants that are represented by sequencing reads both in tumor and its mat- 
ched normal samples are considered. Optimization methods/workflows have been 
designed to analytically access the best combination of tools (both alignment and 
variant calling) to increase the sensitivity of variant detection [23]. The sensitivity 
of the alignment and variant calling tools are usually assessed by a set of metrics 
like aligner and variant caller-specific base quality plots of the variants called, 
transition/transversion (Ti/Tv) ratios, and SNP rediscovery rate using microarrays 
[23]. Further cross-contamination in tumor samples are assessed using tools like 
ContEst [24]. Searching variants against the known cancer-specific variants in 
databases like COSMIC [25-27] is the first step to find out whether the variant/gene 
is unique/novel or have been found in the same or other cancer types previously. 
There are cancer-specific tools to perform annotation and functional analysis. The 
common annotation tools are ANNOVAR [28] and VEP [29]. CRAVAT [30] 
provides predictive scores for different types of variants (both somatic and germ- 
line) and annotations from published literature and databases. It uses a specific 
cancer database with the CHASM analysis option. Genes with a CHASM score of a 
certain value are considered significant for comparison with other functional 
analyses. IntoGen [31], MutSigCV [32], and MuSiC2 [33] are other tools that are 
used for annotation and functional analyses of somatic variants. 


7 Data Validation, Visualization, and Interpretation 


Once annotated, the cancer-specific genes are validated in the same discovery set 
and also using a larger set of validation samples. Validation is largely done using 
either an orthologous sequencing method/chemistry, mass-spec-based mutation 
detection methods, and/or using Sanger sequencing technique. Finally the validated 
variants are mapped to pathways using tools like Graphite Web [34] that employs 
both the topological and multivariate pathway analyses with an interactive network 
for data visualizations. Once the network of genes is obtained, the interactions are 
drawn using tools like CytoScape [35-37]. Variants can also be visualized by using 
Circos [38], a cancer-specific portal like cbio portal [39] or with a viewer like the 
integrative genomics viewer (IGV) [40]. Finally, the genes that are altered in a 
specific cancer tissue are validated using functional screening methods using 
specific gene knockouts to understand their function and relationship with other 
genes. 
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8 High-Throughput Data on Human Cancers 


The current projects on cancer genomics are aimed to produce a large amount of 
sequence information as primary output and information on variant data (somatic 
mutations, insertions and deletions, copy number variations and other structural 
variations in the genome). In order to analyze the large amount of data, 
high-performance compute clusters (HPC) with large memory and storage capacity 
are required. Additionally, higher frequency, high-throughput multi-core chips 
along with the ability to do high-volume data analysis in memory are often 
required. Due to the sheer number of files, and not just the size of the files, that need 
to be processed, the read/write capability is an important parameter for sequence 
analysis. For effective storage and analysis of sequencing and related metadata, 
network access storage systems, providing file-level access, are recommended. 
Additionally, there is a need for an effective database for data organization for easy 
access, management, and data update. Several data portals, primarily made by the 
large consortia are developed. Prominent among them are: The Cancer Genome 
Atlas (TCGA, https://tcga-data.nci.nih.gov/tcga/) data portal; cbio data portal [39] 
(developed at the Memorial Sloan-Kettering Cancer Center, http://www.cbioportal. 
org); the International Cancer Genome Consortium (ICGC) data portal (https://dcc. 
icgc.org); and the Sanger Institute’s Catalogue of Somatic Mutations in Cancer 
(COSMIC) database [25] portal (http://cancer.sanger.ac.uk/cosmic). 

Although biological databases are created using many different platforms, the 
most common among them are MySQL and Oracle. MySQL is more popular 
database because of its open source. Although the consortia-led efforts (like TCGA 
and ICGC) have resulted in large and comprehensive databases covering most 
cancer types, the sites are not user-friendly and do not accept external data for 
integration and visualization. Therefore, efforts like cbio portal (http://www. 
cbioportal.org) are required to integrate data and user-friendly data search and 
retrieval. However, such efforts have to balance keeping in mind the cost and time 
required versus usability and additional value addition from the new database. The 
common databases use software systems known as Relational Database Manage- 
ment Systems (RDBMS) that use SQL (Structured Query Language) for querying 
and maintaining the databases. MySQL is a widely used open source RDBMS. 
Although most biological database uses MySQL or other RDBMS, it has its lim- 
itations as far as large data is concerned. First, big data is assumed to come in 
structured, semi-structured, and unstructured manner. Second, traditional SQL 
databases and other RDBMS lack ability to scale out a requirement for databases 
containing large amount of data. Third, RDBMS cannot scale out with inexpensive 
hardware. All these make RDBMS unsuitable for large data uses. This is primarily 
filled by other databases like NoSQL that are document-oriented graph databases 
that are non-relational, friendly to HPC environment, schema-less, and built to scale 
[41]. One of the important parameters in a database is the ability to take care of 
future increase in data size and complexity (Fig. 3), therefore having an ability to 
scale in both these parameters. Although it is a good idea to think of databases that 
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Fig. 3 Two important parameters of big data and the place for an ideal database 


have the ability to scale out, and accommodate variety and volume of future data 
increase, due to simplicity and ease of use, most small labs stick with MySQL 
database that uses variety of data, commonly used middleware and web server, and 
browser for data retrieval and visualization. 


9 Large-Scale Cancer Genome Projects 


Advances in technology have fuelled interest in the cancer research community that 
has resulted in several large publicly funded consortia-based efforts to catalogue 
changes in primary tumors of various types. Some of the notable and prominent 
efforts in this direction are, The Cancer Genome Atlas (TCGA) project (http://www. 
cancergenome.nih.gov/), the International Cancer Genome Consortium (ICGC) 
project (https://icgc.org) [42], the Cancer Genome Project (http://www.sanger.ac. 
uk/genetics/CGP/), and the Therapeutically applicable Research to Generate 
Effective Treatments (http://target.cancer.gov/) project. The National Cancer Insti- 
tute (NCI) and the National Human Genome Research Institute (NHGRI) of USA 
initially launched the TCGA as a pilot project in 2006 even before the first human 
resequencing work using HTS platforms was published. The TCGA effort plans to 
produce a comprehensive understanding of the molecular basis of cancer and 
currently has grown to include samples from more than 11,000 patients across 33 
different cancer types. The ICGC is an international consortium that plans to obtain 
a comprehensive description of various molecular changes (genomic, transcrip- 
tomic and epigenomic) in 50 different tumor types and/or subtypes. ICGC currently 
has participants from 18 countries studying cancer samples from more than 12000 
donors on 21 tumor types. All the consortia projects are producing substantial 
resource for the wider cancer research community. 

Till date, HTS data on several cancer types have been generated and data analysis 
confirmed the presence of somatic mutations in important genes, significant changes 
in gene/miRNA expression, hyper- and hypo-methylation in gene promoters, and 
structural variations in the cancer genomes [32, 43—66]. Additionally, comparative 
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analysis of different analytical tools have been published for cancer data analysis 
[23, 67-83]. Pan-cancer analyses projects have also have come up with specific and 
shared regions among different cancer types [32, 65, 84—88]. 


10 Cancer Research-Specific Challenges 


There are challenges related to HTS assays using tumor tissues. For a HTS assay to 
have clinical utility, several challenges need to be overcome. The challenges can be 
clinical, technical, biological, statistical, regulatory, and market-related and are 
outlined in Table 2. 

Clinical challenges: First among the clinical challenges is related to sample 
quantity. For retrospective studies to be meaningful, assays must be robust to use 
nucleic acids derived from formalin-fixed paraffin-embedded (FFPE) tissues. Tissue 
sections extracted are not often big to yield sufficient quantity of nucleic acids that 
can be used for sequencing, and validation studies even with the newer assays that 


Table 2 Challenges of making high-throughput assays, especially sequencing-based assays, 
meaningful in clinics 


Type of Issues 
challenge 
Clinical e Sample quantity from tissue biopsies 


e Sample quality (FFPE tissue, chemical-crosslinking, etc.) 
e Tumor heterogeneity 


Biological e Background somatic mutation (needle in a haystack problem) 
e Varying mutation rate 
e Finding the right matched normal 
e Lack of proper controls for data assessment 


Technical e Sequencing error rate 
e Repeats in the genome 
e Read length 
e Stretch of homopolymers in the genome 
e Regions of low-complexity in the genome 
e Nucleotide composition 


Statistical e Large sample number requirement to catch somatic mutations at low 
frequency 


Regulatory e Lack of standards for clinical evaluation 
e Lack of controls to judge sequencing and variant calling error and 
accuracy 
e Lack of proper regulatory guidelines 


Market-related e Price 
e Competition 
e Acceptability 
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use only tens of nanograms of nucleic acids as starting material. Even if one 
manages to get enough nucleic acids from the FFPE tissues, the quality of nucleic 
acids extracted is not the best of quality and is often fragmented. Additionally, 
chemical modifications like the presence of cross-links and depurination and the 
presence of certain impurities in the FFPE-extracted DNA make them less amen- 
able to alterations required for high-throughput assays. Therefore, FFPE-extracted 
DNA can have a stronger influence on the HTS assays. Cancer tissues are 
heterogeneous [32, 89] and in certain cases extremely heterogeneous (for example 
in pancreatic adenocarcinoma) that cautions overinterpreting HTS data from a lump 
of tumor tissue as shown in metastatic renal-cell carcinoma [90]. Therefore, in 
heterogenous tumors, the mutational burden may be underestimated. Studying 
such intra-tumor heterogeneity may aid the case for combination therapeutic 
approaches in cancer [91]. Analytical methods have been devised in the past to 
detect tumor heterogeneity [73, 92, 93]. 

Biological challenges: The next challenge is biological where finding somatic 
mutations, especially, those present at very low frequency, among the sea of normal 
background is really difficult. The use of a matched normal sample for cancer 
sequencing is essential to find somatic variants but, at times, the matched normal 
tissue might be hard to get and therefore, variants found in lymphocytes from the 
same patients are often used as normal samples. Another problem in sequencing 
tumor tissue DNA is cross-contamination. Analytical tools have been developed to 
detect the level of cross-contamination in tumor tissues from both sequencing and 
array data [24, 94]. To overcome both the heterogeneity and the 
cross-contamination issue, the best way is to perform DNA/RNA sequencing 
derived from a single tumor cell. Single-cell genomics is likely to help and improve 
detection, progression, and prediction of therapeutic efficacy of cancer [95]. Several 
reports have been published on single-cell sequencing of different cancers and 
analytical tool development to analyze data from a single tumor cell [96—108]. 
Although the problem of heterogeneity is overcome with single-cell sequencing, the 
fundamental questions may still linger, 1.e., how many single cells have to be 
sequenced and if the signature is different in different single tumor cells. Addi- 
tionally, there are limitations to the current protocols for isolation of single tumor 
cells and the inaccuracies involved in whole genome amplification of genomic 
DNA derived from a single cell. Therefore, capturing minute amounts of genetic 
material and amplifying them remain as one the greatest challenges in single cell 
genomics [109, 110]. 

Technical challenges: The third type of challenge is related to technical issues 
with current generation of sequencing instruments. Depending on the instrument in 
use, there could be an issue related to high error rate, length of the read, 
homopolymer stretches, and GC-rich regions in the genome. Additionally, accurate 
sequencing and assembling correct haplotype structures for certain regions of the 
genome, like the human leukocyte antigen (HLA) region, are challenging due to 
shorter read lengths generated in second generation DNA sequencers, presence of 
polymorphic exons and pseudogenes, and repeat rich region. 
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Statistical challenges: One of the biggest challenges to find driver mutations in 
cancer is related to sample number. Discovering rare driver mutations in cancer is 
extremely challenging, especially when sample numbers are not adequate. This, 
so-called, “the long tail phenomenon” is quite common in many of the cancer 
genome sequencing studies. Discovering rare driver mutations (found at 2 % fre- 
quency or lower) requires sequencing a large number of samples. For example, in 
head and neck cancer, imputations have shown that it will take 2000 tumor:normal 
samples to be sequenced at 90 % power in 90 % of the genes to find somatic 
variants present at 2 % frequency or higher [43]. 

Regulatory and other challenges: In order for cancer personalized medicine to 
become a reality, proper regulatory and policy framework need to be in place. 
Issues around how to deal with germline changes along with strict and proper assay 
and technical controls/standards are needed to be in place to assess biological, 
clinical, and technical accuracy and authenticity. A great beginning in this direction 
has already been made by the genome in a bottle consortium (https://sites.stanford. 
edu/abms/giab) hosted by the National Institute of Standards and Technology of the 
USA that has come up with reference materials (reference standards, reference 
methods, and reference data) to be used in sequencing. Finally, in order for cutting 
edge genomic tests to become a reality, collaboration and cooperation between 
academic centers and industry are absolutely necessary [111]. Additionally, 
acceptability criteria and proper pricing control mechanism(s) need to be in place by 
the government. This is necessary for countries like India where genomic tests are 
largely unregulated. 


11 Conclusion 


Cancer research has changed since the introduction of technologies like DNA 
microarray and high-throughput sequencing. It is now possible to get a 
genome-wide view on a particular tumor rather than looking at a handful of genes. 
The biggest challenge for finding actionable variants in cancer remains at the level 
of data analysis and understanding of their functional importance. Recent demon- 
strations [112—115] of gene editing systems like CRISPR-Cas9 in understanding the 
function of cancer-related genes and their role(s) in carcinogenesis and metastasis 
will play a big role in the future. Further, high-throughput sequencing technology 
can be used to providing information on individual cancer regulome by integrating 
information on genetic variants, transcript variants, regulatory proteins binding to 
DNA and RNA, DNA and protein methylation, and metabolites. Finally, for big 
data to bear fruits in cancer diagnosis, prognosis, and treatment, processes like; 
simplified data analytics platforms; accurate sequencing chemistry; standards for 
measuring clinical accuracy, precision and sensitivity; proper country-specific 
regulatory guidelines and stringent yet ethical framework against data misuse; need 
to be in place [111]. 
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